Original: Distributed Influence-Augmented Local Simulators for Parallel MARL in Large Networked Systems
Revised: Distributed Influence-Augmented Local Simulators for Parallel MARL in Large Networked Systems

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Due to its high sample complexity, simulation is, as of today, critical for thesuccessful application of reinforcement learning.
Revised: Due to its high sample complexity, simulation is, as of today, critical for thesuccessful application of reinforcement learning.

Original: Many real-world problems,however, exhibit overly complex dynamics, which makes their full-scale simulationcomputationally slow.
Revised: Many real-world problems,however, exhibit overly complex dynamics, which makes their full-scale simulationcomputationally slow.

Original: In this paper, we show how to decompose large networkedsystems of many agents into multiple local components such that we can buildseparate simulators that run independently and in parallel.
Revised: In this paper, we show how to decompose large networkedsystems of many agents into multiple local components such that we can buildseparate simulators that run independently and in parallel.

Original: To monitor the influencethat the different local components exert on one another, each of these simulators isequipped with a learned model that is periodically trained on real trajectories.
Revised: To monitor the influencethat the different local components exert on one another, each of these simulators isequipped with a learned model that is periodically trained on real trajectories.

Original: Ourempirical results reveal that distributing the simulation among different processesnot only makes it possible to train large multi-agent systems in just a few hours butalso helps mitigate the negative effects of simultaneous learning
Revised: Ourempirical results reveal that distributing the simulation among different processesnot only makes it possible to train large multi-agent systems in just a few hours butalso helps mitigate the negative effects of simultaneous learning

Original:  Introduction
Revised:  Introduction

Original: Imagine we have to train a team of agents in controlling the traffic lights of a very large city, solarge that we simply cannot control all traffic lights using a single policy.
Revised: Imagine we have to train a team of agents in controlling the traffic lights of a very large city, solarge that we simply cannot control all traffic lights using a single policy.

Original: The first step would beto split the problem into multiple sub-regions.
Revised: The first step would beto split the problem into multiple sub-regions.

Original: A natural division would be to assign one trafficlight to each agent.
Revised: A natural division would be to assign one trafficlight to each agent.

Original: Then, since the agents act locally, we would limit their observations to containonly local information.
Revised: Then, since the agents act locally, we would limit their observations to containonly local information.

Original: This partial observability could affect their optimal policies but would alsomake each individual decision-making problem more manageable (McCallum, 1995; Dearden and
Revised: This partial observability could affect their optimal policies but would alsomake each individual decision-making problem more manageable (McCallum, 1995; Dearden and

Original: Boutilier, 1997).
Revised: Boutilier, 1997).

Original: Moreover, we may also want to reward agents only for what occurs in their localneighborhood such that we reduce the variance of the returns (Spooner et al., 2021) and facilitatecredit assignment (Castellini et al., 2020).
Revised: Moreover, we may also want to reward agents only for what occurs in their localneighborhood such that we reduce the variance of the returns (Spooner et al., 2021) and facilitatecredit assignment (Castellini et al., 2020).

Original: Finally, we could train all agents together on a big trafficsimulator that reproduces the global dynamics.
Revised: Finally, we could train all agents together on a big trafficsimulator that reproduces the global dynamics.

Original: However, if the city is truly large, it could take weeksor even months to optimize their policies.
Revised: However, if the city is truly large, it could take weeksor even months to optimize their policies.

Original: That is assuming training actually converges.
Revised: That is assuming training actually converges.

Original: One may argue that, since the agents’ observations and rewards are local, we could as well train themon separate simulators that model only the local transition dynamics (i.e. cars moving within eachof the sub-regions; van der Pol and Oliehoek 2016).
Revised: One may argue that, since the agents’ observations and rewards are local, we could as well train themon separate simulators that model only the local transition dynamics (i.e. cars moving within eachof the sub-regions; van der Pol and Oliehoek 2016).

Original: This approach might work if the agents’ localtransitions are isolated from the rest of the system (Becker et al., 2003), but would probably breakwhen the local regions are coupled.
Revised: This approach might work if the agents’ localtransitions are isolated from the rest of the system (Becker et al., 2003), but would probably breakwhen the local regions are coupled.

Original: This is because the local simulators would fail to account for thefact that the agents’ local regions belong in a larger system and depend on one another.
Revised: This is because the local simulators would fail to account for thefact that the agents’ local regions belong in a larger system and depend on one another.

Original: A solution is to model the influence the global system exerts on each of the local regions.
Revised: A solution is to model the influence the global system exerts on each of the local regions.

Original: Fortunately,this does not necessarily imply modelling the entire system, or else we would just use the globalsimulator.
Revised: Fortunately,this does not necessarily imply modelling the entire system, or else we would just use the globalsimulator.

Original: In many scenarios, such as in the traffic problem, even though the local regions may beaffected by many external variables (e.g.traffic densities in other parts of the city), they are onlydirectly influenced by a small subset of them (e.g. road segments that connect the intersections withthe rest of the city).
Revised: In many scenarios, such as in the traffic problem, even though the local regions may beaffected by many external variables (e.g.traffic densities in other parts of the city), they are onlydirectly influenced by a small subset of them (e.g. road segments that connect the intersections withthe rest of the city).

Original: This subset of variables are known as the influence sources.
Revised: This subset of variables are known as the influence sources.

Original: The theoreticalframework of Influence-Based Abstraction (Oliehoek et al., 2021) shows that by monitoring the posterior distribution of the influence sources given the action local state history (ALSH) one cansimulate realistic trajectories that match those produced by the global simulator.
Revised: The theoreticalframework of Influence-Based Abstraction (Oliehoek et al., 2021) shows that by monitoring the posterior distribution of the influence sources given the action local state history (ALSH) one cansimulate realistic trajectories that match those produced by the global simulator.

Original: The resultingsimulator, known as the influence-augmented local simulator (IALS), has been proven effective insingle agent scenarios when combined with planning (He et al., 2020) and reinforcement learning(RL) algorithms (Suau et al., 2022).
Revised: The resultingsimulator, known as the influence-augmented local simulator (IALS), has been proven effective insingle agent scenarios when combined with planning (He et al., 2020) and reinforcement learning(RL) algorithms (Suau et al., 2022).

Original: In this paper we extend the IBA framework to multi-agent domains.
Revised: In this paper we extend the IBA framework to multi-agent domains.

Original: We show how to decomposelarge networked systems, such as the previous traffic example, into multiple sub-regions so that wecan replace the GS by a distributed network of IALSs that can run independently and in parallel.
Revised: We show how to decomposelarge networked systems, such as the previous traffic example, into multiple sub-regions so that wecan replace the GS by a distributed network of IALSs that can run independently and in parallel.

Original: There is one important caveat to this.
Revised: There is one important caveat to this.

Original: The IBA framework assumes there is only a single agentlearning at a time.
Revised: The IBA framework assumes there is only a single agentlearning at a time.

Original: This assumption is needed to make the influence distributions stationary.
Revised: This assumption is needed to make the influence distributions stationary.

Original: Thisimplies that in our case, since we want the agents to learn simultaneously, previously computedinfluence distributions would no longer be valid after the agents update their policies.
Revised: Thisimplies that in our case, since we want the agents to learn simultaneously, previously computedinfluence distributions would no longer be valid after the agents update their policies.

Original: The naivesolution would be to recompute new influence distributions every time any of the agents updates itspolicy.
Revised: The naivesolution would be to recompute new influence distributions every time any of the agents updates itspolicy.

Original: However, we argue that this is not only impractical, since recomputing the distributions is notwithout costs, but also undesirable.
Revised: However, we argue that this is not only impractical, since recomputing the distributions is notwithout costs, but also undesirable.

Original: The theoretical results in Section 4.1 demonstrate that multiple(similar) joint policies may induce the same influence distributions, and that even when they vary alittle, they can still elicit the same optimal policies.
Revised: The theoretical results in Section 4.1 demonstrate that multiple(similar) joint policies may induce the same influence distributions, and that even when they vary alittle, they can still elicit the same optimal policies.

Original: Further, our insights in Section 4.3 hint that whatseems to be a problem at first, may in fact be an advantage, since in many situations, maintaining theprevious influence distributions implies that the local transitions, although biased, remain stationary.
Revised: Further, our insights in Section 4.3 hint that whatseems to be a problem at first, may in fact be an advantage, since in many situations, maintaining theprevious influence distributions implies that the local transitions, although biased, remain stationary.

Original: Contributions The main contributions of this paper are: (1) adapting IBA to multi-agent reinforcement learning (MARL),1 and demonstrating that simultaneous learning is possible withoutincurring major computational costs, (2) showing that by distributing the simulation among differentprocesses we can parallelize training and scale up to systems with many agents, (3) revealing that thenon-stationarity issues inherent to MARL are partly mitigated as a result of this training scheme.
Revised: Contributions The main contributions of this paper are: (1) adapting IBA to multi-agent reinforcement learning (MARL), and demonstrating that simultaneous learning is possible without incurringmajor computational costs, (2) showing that by distributing the simulation among different processes we can parallelize training and scale up to systems with many agents, (3) revealing that thenon-stationarity issues inherent to MARL are partly mitigated as a result of this training scheme.

Original:  Related Work
Revised:  Related Work

Original: A few prior works have investigated the computational benefits of factorizing large systems intoindependent local regions (Nair et al., 2005; Varakantham et al., 2007; Kumar et al., 2011; Witwickiand Durfee, 2011).
Revised: A few prior works have investigated the computational benefits of factorizing large systems intoindependent local regions (Nair et al., 2005; Varakantham et al., 2007; Kumar et al., 2011; Witwickiand Durfee, 2011).

Original: Unfortunately, since local regions are often coupled to one another, such factorizations are not always appropriate.
Revised: Unfortunately, since local regions are often coupled to one another, such factorizations are not always appropriate.

Original: Nonetheless, in many cases, the interactions between regionsoccur through a limited number of variables.
Revised: Nonetheless, in many cases, the interactions between regionsoccur through a limited number of variables.

Original: Using this property, the theoretical work by Oliehoeket al.
Revised: Using this property, the theoretical work by Oliehoeket al.

Original: (2021) on influence-based abstraction (IBA) describes how to build influence-augmented localsimulators (IALS) of local-POMDPs, which model only the variables in the environment that aredirectly relevant to the agent while monitoring the response of the rest of the system with the influencepredictor.
Revised: (2021) on influence-based abstraction (IBA) describes how to build influence-augmented localsimulators (IALS) of local-POMDPs, which model only the variables in the environment that aredirectly relevant to the agent while monitoring the response of the rest of the system with the influencepredictor.

Original: The problem is that the exact computation of the conditional influence distribution isintractable and we can only try to estimate it from data.
Revised: The problem is that the exact computation of the conditional influence distribution isintractable and we can only try to estimate it from data.

Original: Congeduti et al.
Revised: Congeduti et al.

Original: (2021) provide theoreticalbounds on the value loss when planning with approximate influence predictors.
Revised: (2021) provide theoreticalbounds on the value loss when planning with approximate influence predictors.

Original: The work by Heet al.
Revised: The work by Heet al.

Original: (2020) has empirically demonstrated the advantage of this approach to improve the efficiency ofonline planning in two discrete toy problems.
Revised: (2020) has empirically demonstrated the advantage of this approach to improve the efficiency ofonline planning in two discrete toy problems.

Original: Suau et al.
Revised: Suau et al.

Original: (2022) scale the method to high-dimensionalproblems by integrating the IBA framework with single-agent RL showing that the IALS can trainpolicies much faster than the GS.
Revised: (2022) scale the method to high-dimensionalproblems by integrating the IBA framework with single-agent RL showing that the IALS can trainpolicies much faster than the GS.

Original: In this paper, we extend the IBA solution to MARL and explainhow to build a network of independent IALS such that we can train agents in parallel.
Revised: In this paper, we extend the IBA solution to MARL and explainhow to build a network of independent IALS such that we can train agents in parallel.

Original: One of the consequences of training agents on independent simulators is that the non-stationarityissues arising from having the agents learn simultaneously are partly mitigated.
Revised: One of the consequences of training agents on independent simulators is that the non-stationarityissues arising from having the agents learn simultaneously are partly mitigated.

Original: There is a sizeablebody of literature that concentrates on this issue (Hernandez-Leal et al., 2017), we include a reviewof these works in Appendix B for completeness.
Revised: There is a sizeablebody of literature that concentrates on this issue (Hernandez-Leal et al., 2017), we include a reviewof these works in Appendix B for completeness.

Original: However, we note that the main purpose of thispaper is to scale MARL up to systems with many agents.
Revised: However, we note that the main purpose of thispaper is to scale MARL up to systems with many agents.

Original: Hence, we are not concerned here withcomparing our method with those that exclusively target non-stationarity, especially given that, forscalability reasons, these cannot be applied to the high-dimensional problems we consider here.
Revised: Hence, we are not concerned here withcomparing our method with those that exclusively target non-stationarity, especially given that, forscalability reasons, these cannot be applied to the high-dimensional problems we consider here.

Original:  Preliminaries
Revised:  Preliminaries

Original: The type of problems we describe in the introduction can be formulated as factored partially observablestochastic games (Hansen et al., 2004), which are defined as follows.
Revised: The type of problems we describe in the introduction can be formulated as factored partially observablestochastic games (Hansen et al., 2004), which are defined as follows.

Original: Definition 1 (fPOSG) .
Revised: Definition 1 (fPOSG) .

Original: A factored partially observable stochastic game (fPOSG) is a tuple⟨ N, S,  A, T, { R i } , Ω , { O i }⟩ where N = { 1 , ..., n } is the set of n agents, S is the set of jstate variablesS
Revised: A factored partially observable stochastic game (fPOSG) is a tuple⟨ N, S, F, A, T, { R i } , Ω , { O i }⟩ where N = { 1 , ..., n } is the set of n agents, S is the set state variables

Original:  = { S 1 , ..., S k } , such that every state s t ∈ × kj =1 S j is a k -dimensional vectors t = ⟨ s 1 ,t , ..., s k,t ⟩ ,A = × i ∈
Revised: S = { S 1 , ..., S k } , such that every state s t ∈ × kj =1 S j is a k -dimensional vector s t = ⟨ s 1 ,t , ..., s k,t ⟩ ,

Original:  N A
Revised: A = × i ∈ N A

Original: i is the set of joint actions a t = ⟨ a t 1 , ..., a tn ⟩ , with A i being the setof actions for agent i , T is the transition function, with T ( s t +1 | s t , a t ) , R i ( s t , a ti ) is the immediatereward for agent i , Ω = × i ∈ N Ω
Revised: i is the set of joint actions a t = ⟨ a t 1 , ..., a tn ⟩ , with A i being the set of actions for agenti , T is the transition function, with T ( s t +1 | s t , a t ) , R i ( s t , a ti ) is the immediate reward for agent i ,Ω = × i ∈ N Ω

Original: i is the set of joint observations o ti = ⟨ o t 1 , ..., o tn ⟩ , with Ω i being theset of observations for agent i , and O i is the observation function for agent
Revised: i is the set of joint observations o ti = ⟨ o t 1 , ..., o tn ⟩ , with Ω i being the set of observationsfor agent i , and O i is the observation function for agent

Original: i , O i
Revised: i , O i

Original: ( o ti | s t ) .
Revised: ( o ti | s t ) .

Original: Solving the fPOSG implies finding the policy
Revised: Solving the fPOSG implies finding the policy

Original: π i for each agent i that maximizes the expected return
Revised: π i for each agent i that maximizes the expected return

Original: G t ; as defined in Sutton and Barto (1998).
Revised: G t ; as defined in Sutton and Barto (1998).

Original: However, agents receive only partial observations
Revised: However, agents receive only partial observations

Original: o i ofthe true state s , which are not necessarily Markovian.
Revised: o i ofthe true state s , which are not necessarily Markovian.

Original: Therefore, optimal policies are in generalhistory-dependent in a POSG.
Revised: Therefore, optimal policies are in generalhistory-dependent in a POSG.

Original: Hence, we define the agents’ policies π i ( a ti | h ti ) as mappings fromaction-observation histories (AOH), h ti =
Revised: Hence, we define the agents’ policies π i ( a ti | h ti ) as mappings fromaction-observation histories (AOH), h ti =

Original: ⟨ o 1 i , a 1 i , ..., a t − 1 i , o ti ⟩ , to probability distributions overactions, such that agent i ’s optimal policy π ∗ i is the one that for every AOH h ti selects the action withthe highest Q -value Q π ∗ i ( h t i , a t i ) =
Revised: ⟨ o 1 i , a 1 i , ..., a t − 1 i , o ti ⟩ , to probability distributions overactions, such that agent i ’s optimal policy π ∗ i is the one that for every AOH h ti selects the action withthe highest Q -value Q π ∗ i ( h ti , a ti ) =

Original: E [ G t | h t i , a t i , π ∗ i ] .
Revised: E [ G t | h ti , a ti , π ∗ i ] .

Original: Given the structural assumptions we made in the introduction about the agents’ only being ableto observe and be rewarded for what occurs in their local neighborhood, we can narrow down theproblem formulation and work with a specific class of fPOSGs called local-form fPOSGs (Oliehoeket al., 2021), which better encompass the problems we consider here.
Revised: Given the structural assumptions we made in the introduction about the agents’ only being able [Figure] to observe and be rewarded for what occurs in their local neighborhood, we can narrow down theproblem formulation and work with a specific class of fPOSGs called local-form fPOSGs (Oliehoeket al., 2021), which better encompass the problems we consider here.

Original: Definition 2 (Local-form fPOSG) .
Revised: Definition 2 (Local-form fPOSG) .

Original: A Local-form fPOSG is a fPOSG where O i and R i dependonly on a subset of m state variables X i =
Revised: A Local-form fPOSG is a fPOSG where O i and R i dependonly on a subset of m state variables X i =

Original: { X 1 i , ..., X mi } ⊆ S , with m ≤ k (number of statevariables), and agent i ’s local states x i ∈ ×
Revised: { X 1 i , ..., X mi } ⊆ S , with m ≤ k (number of statevariables), and agent i ’s local states x i ∈ ×

Original: mj
Revised: mj

Original: =1 X ji being vectors x ti = ⟨ x 1 ,ti , ..., x k,ti ⟩ , such that
Revised: =1 X ji being vectors x ti = ⟨ x 1 ,ti , ..., x k,ti ⟩ , such that

Original: O i ( o ti | s t )
Revised: O i ( o ti | s t )

Original: = ˙
Revised: = ˙

Original: O i ( o ti | x ti ) and R i ( s t , a ti )
Revised: O i ( o ti | x ti ) and R i ( s t , a ti )

Original: = ˙
Revised: = ˙

Original: R i ( x ti , a ti ) , where ˙
Revised: R i ( x ti , a ti ) , where ˙

Original: O i and ˙
Revised: O i and ˙

Original: R i are the local observation andreward functions for agent i .
Revised: R i are the local observation andreward functions for agent i .

Original: 3.1 Influence-Based Abstraction
Revised: 3.1 Influence-Based Abstraction

Original: We now describe the IBA framework (Oliehoek et al., 2021) which intends to simplify the local-formfPOSG formulation by exploiting its structural properties.
Revised: We now describe the IBA framework (Oliehoek et al., 2021) which intends to simplify the local-formfPOSG formulation by exploiting its structural properties.

Original: The framework assumes that there isa single agent i learning at a time while all other agents’ polices π − i are fixed.
Revised: The framework assumes that there isa single agent i learning at a time while all other agents’ polices π − i are fixed.

Original: Hence, from theperspective of agent i the problem reduces to a POMDP (Kaelbling et al., 1996) where states arepairs ⟨ s t , h t − i ⟩ , with h t − i being the AOHs of all agents but agent i (Nair et al., 2003).
Revised: Hence, from theperspective of agent i the problem reduces to a POMDP (Kaelbling et al., 1996) where states are pairs⟨ s t , h t − i ⟩ , with h t − i being the AOHs of all agents but agent i (Nair et al., 2003).

Original: Finding alocally optimal solution to the Local-form fPOSG can be approached by‘alternating maximization’or ‘coordinate ascent’. That is, sequentially iterating over all agents, possibly multiple times, andsolving their respective POMDPs (Nair et al., 2003; Oliehoek and Amato, 2016).
Revised: Finding a solution tothe Local-form fPOSG can be approached by sequentially iterating over all agents, possibly multipletimes, and solving their respective POMDPs.

Original: Looking at the definition of Local-form fPOSG, one can argue that, when solving for agent i ,sampling actions from the policies of all the other agents π − i ( a t − i | h t − i ) and simulating the transitions
Revised: Looking at the definition of Local-form fPOSG, one can argue that, when solving for agent i ,sampling actions from the policies of all the other agents π − i ( a t − i | h t − i ) and simulating the transitions

Original: T ( s t +1 | s t , a t ) of the full set of state variables is unnecessary, and while doing so is possible insmall problems, it might become computationally intractable in large domains with many agents.
Revised: T ( s t +1 | s t , a t ) of the full set of state variables is unnecessary, and while doing so is possible insmall problems, it might become computationally intractable in large domains with many agents.

Original: Instead, we can define a new transition function ¯ T i that models only agent
Revised: Instead, we can define a new transition function ¯ T i that models only agent

Original: i ’s local state variablesx i , ¯ T i ( x i,t +1 | x ti , a ti ) .
Revised: i ’s local state variablesx i , ¯ T i ( x i,t +1 | x ti , a ti ) .

Original: The problem is that x t +1 i may still depend on the other agents actions a t − i andthe non-local state variables S \ X i , which means that ¯ T i is not well defined.
Revised: The problem is that x t +1 i may still depend on the other agents actions a t − i andthe non-local state variables S \ X i , which means that ¯ T i is not well defined.

Original: Fortunately, in manyproblems, only a fraction of the non-local state variables will directly influence agent i ’s local region.
Revised: Fortunately, in manyproblems, only a fraction of the non-local state variables will directly influence agent i ’s local region.

Original: The diagram on the left of Figure 1 is a Dynamic Bayesian Network (DBN) (Pearl, 1988; Boutilieret al., 1999) describing a particular instance of the transition dynamics for a generic agent i ina local-fPOSG.
Revised: The diagram on the left of Figure 1 is a Dynamic Bayesian Network (DBN) (Pearl, 1988; Boutilieret al., 1999) describing a particular instance of the transition dynamics for a generic agent i ina local-fPOSG.

Original: Agent i ’s local region, corresponds to the variables that lie within the red box,x i ∈
Revised: Agent i ’s local region, corresponds to the variables that lie within the red box,x i ∈

Original: X i =
Revised: X i =

Original: { X 1 i , X 2 i } .
Revised: { X 1 i , X 2 i } .

Original: The diagram also shows the non-local variables, known as influence sourcesu i ∈
Revised: The diagram also shows the non-local variables, known as influence sourcesu i ∈

Original: U
Revised: U

Original: i ⊆
Revised: i ⊆

Original: S \ X i , that influence the local region directly.
Revised: S \ X i , that influence the local region directly.

Original: The three dots on the top indicate that therecan be, potentially many, other non-local variables in S affecting the local variables
Revised: The three dots on the top indicate that therecan be, potentially many, other non-local variables in S affecting the local variables

Original: X i .
Revised: X i .

Original: These aredenoted by
Revised: These aredenoted by

Original: y i ∈ Y i ⊆
Revised: y i ∈ Y i ⊆

Original: S \ X
Revised: S \ X

Original: i ∪ U i .
Revised: i ∪ U i .

Original: The diagram also shows that agent j can affect agent i ’s local regionthrough its actions a j ∈ A j .
Revised: The diagram also shows that agent j can affect agent i ’s local regionthrough its actions a j ∈ A j .

Original: However, both Y i and A j can only influence X i via U i .
Revised: However, both Y i and A j can only influence X i via U i .

Original: Hence, given u ti ,x t +1{++
+} i is conditionally independent of y ti and a tj , therefore P ( x t +1 i | x ti , u ti , y ti , a tj ) =
Revised: Hence, given u t i ,x t {++
+} i is conditionally independent of y ti and a tj , therefore P ( x t +1 i | x ti , u ti , y ti , a tj ) =

Original: P ( x t +1 i | x ti , u ti ) .[Figure]
Revised: P ( x t +1 i | x ti , u ti ) .

Original: The above implies that by inferring the value of the influence source u ti , we can monitor the influenceof the other agents and the non-local state variables and thus compute the local state transitions.
Revised: The above implies that by inferring the value of the influence source u ti , we can monitor the influenceof the other agents and the non-local state variables and thus compute the local state transitions.

Original: Thiscan be done by keeping track of the action-local-state history (ALSH) l ti = ⟨ x 1 i , a 1 i ..., a t − 1 i , x ti ⟩ . 
Revised: Thiscan be done by keeping track of the action-local-state history (ALSH) l ti = ⟨ x 1 i , a 1 i ..., a t − 1 i , x ti ⟩ . [Figure]

Original: Definition 3 (IALM) .
Revised: Definition 3 (IALM) .

Original: An influence-augmented local Model (IALM) for agent i is a tuple⟨ X i ,
Revised: An influence-augmented local Model (IALM) for agent i is a tuple⟨ X i ,

Original: U i , A i , ˙ T i , ˙
Revised: U i , A i , ˙ T i , ˙

Original: R i , Ω i , ˙
Revised: R i , Ω i , ˙

Original: O i , I i ⟩ , with local states x
Revised: O i , I i ⟩ , with local states x

Original: i ∈ ×
Revised: i ∈ ×

Original: j ∈| X i | X ji , influence sources u i ∈ ×
Revised: j ∈| X i | X ji , influence sources u i ∈ ×

Original: j ∈| U i | U ji ,local transition function ˙
Revised: j ∈| U i | U ji ,local transition function ˙

Original: T i ( x t +1 i | x ti , u ti , a ti ) , local observation function ˙
Revised: T i ( x t +1 i | x ti , u ti , a ti ) , local observation function ˙

Original: O i ( o t +1 i | x t +1 i ) ,
Revised: O i ( o t +1 i | x t +1 i ) ,

Original: local rewardfunction ˙
Revised: local rewardfunction ˙

Original: R i ( x ti , a ti ) , and influence distribution I i ( u ti | l ti ) .
Revised: R i ( x ti , a ti ) , and influence distribution I i ( u ti | l ti ) .

Original: Using the IALM we can compute agent i ’s local transitions as
Revised: Using the IALM we can compute agent i ’s local transitions as

Original: [Equation]
Revised: [Equation]

Original: Note that, as opposed to the Local-form fPOSG, the transition function ˙
Revised: Note that, as opposed to the Local-form fPOSG, the transition function ˙

Original: T i in the IALM is definedpurely in terms of the local state variables and the influence sources.
Revised: T i in the IALM is definedpurely in terms of the local state variables and the influence sources.

Original: Moreover, since u t d-separates(Bishop, 2006) x t from y t , we only need to maintain a belief over u ti , I i ( u ti | l ti ) , rather than over thefull set of of state variables s t and other agents’ histories h t − i , P ( s t , h t − i | l ti ) .
Revised: Moreover, since u t d-separates(Bishop, 2006) x t from y t , we only need to maintain a belief over u ti , I i ( u ti | l ti ) , rather than over thefull set of of state variables s t and other agents’ histories h t − i , P ( s t , h t − i | l ti ) .

Original: All in all, this translatesinto
Revised: All in all, this translatesinto

Original: a much more compact, yet exact representation of the problem (Oliehoek et al., 2021), whichshould be computationally much lighter than the original Local-form fPOSG.
Revised: a much more compact, yet exact representation of the problem (Oliehoek et al., 2021), whichshould be computationally much lighter than the original Local-form fPOSG.

Original: 3.2 Influence-Augmented Local Simulators
Revised: 3.2 Influence-Augmented Local Simulators

Original: Here we briefly describe how the IALM formulation can be used in practice to build IALS (Suauet al., 2022), which consist of a local simulator and an approximate influence predictor .
Revised: Here we briefly describe how the IALM formulation can be used in practice to build IALS (Suauet al., 2022), which consist of a local simulator and an approximate influence predictor .

Original: Local simulator (LS): The LS is an abstracted version of the environment that only models a smallportion of it.
Revised: Local simulator (LS): The LS is an abstracted version of the environment that only models a smallportion of it.

Original: As opposed to a global simulator (GS), which should closely reproduce the dynamics ofevery state variable, the LS focuses on characterizing the transitions of those variables
Revised: As opposed to a global simulator (GS), which should closely reproduce the dynamics ofevery state variable, the LS focuses on characterizing the transitions of those variables

Original: X i that agent idirectly interacts with, ˙ T i ( x t
Revised: X i that agent idirectly interacts with, ˙ T i ( x t

Original: +1 i | x ti , u ti , a ti ) .
Revised: +1 i | x ti , u ti , a ti ) .

Original: Approximate influence predictor (AIP):
Revised: Approximate influence predictor (AIP):

Original: The AIP monitors the interactions between agenti ’s local region X i , the external variables
Revised: The AIP monitors the interactions between agenti ’s local region X i , the external variables

Original: Y i , and the other agents’ actions A − i , by estimating
Revised: Y i , and the other agents’ actions A − i , by estimating

Original: I i ( u ti | l ti ) .
Revised: I i ( u ti | l ti ) .

Original: Since, due to combinatorial explosion, computing the exact probability I i ( u ti | l ti ) is generallyintractable (Oliehoek et al., 2021), a neural network is used instead to approximate the influencedistribution.
Revised: Since, due to combinatorial explosion, computing the exact probability I i ( u ti | l ti ) is generallyintractable (Oliehoek et al., 2021), a neural network is used instead to approximate the influencedistribution.

Original: Thus, we write ˆ I θ i to denote agent i ’s AIP, where θ i are the network parameters.
Revised: Thus, we write ˆ I θ i to denote agent i ’s AIP, where θ i are the network parameters.

Revised: Due tothe sequential nature of the problem, rather than feeding the full past ALSH every time we make aprediction, a recurrent neural network (RNN) (Hochreiter and Schmidhuber, 1997; Cho et al.

Revised: , 2014)is used such that local states can be processesed one at a time while the ALSH is embedded in the

Revised: RNN’s internal memory.

Revised: The AIP ˆ I

Original: The
Revised: θ

Original: AIP ˆ I θ i is trained on a dataset D i of N samples of the form ( l ti , u ti ) collected from the GS.
Revised: i is trained on a dataset D i of N samples of the form ( l ti , u ti )collected from the GS.

Original: Since roleof the AIP is to estimate the conditional probability of the influence sources u ti given the past ALSH,we can formulate the task as a classification problem and optimize the network using the expectedcross-entropy loss (Bishop, 2006).
Revised: Since role of the AIP is to estimate the conditional probability of the influencesources u ti given the past ALSH, we can formulate the task as a classification problem and optimizethe network using the expected cross-entropy loss (Bishop, 2006).

Original:  Distributed Influence-Augmented Local Simulators
Revised:  Distributed Influence-Augmented Local Simulators

Original: As mentioned in the previous section, local-form fPOSGs are solved iteratively in the IBA framework.
Revised: As mentioned in the previous section, local-form fPOSGs are solved iteratively in the IBA framework.

Original: This means that only a single agent can update its policy at a time.
Revised: This means that only a single agent can update its policy at a time.

Original: Here, we relax this assumptionand discuss the advantages and disadvantages of simultaneous learning.
Revised: Here, we relax this assumption and discuss the advantages and disadvantages of simultaneous learning.

Original: Proofs for all the theoreticalresults in this section can be found in Appendix A.
Revised: Proofs for all the theoreticalresults in this section can be found in Appendix A.

Original: 4.1 Enabling Parallelization
Revised: 4.1 Enabling Parallelization

Original: [Figure]
Revised: [Figure]

Original: {+Environment 3 
+}
Revised: {+Environment 3 
+}

Revised: [Figure]

Revised: Simulators

Original: [Figure]
Revised: [Figure]

Original: The main reason to disallow simultaneous learning is that changes in the other agents’ policies can affect agent i ’s influence distribution
Revised: The main reason to disallow simultaneous learning is that changes in the other agents’ policies can affect agent i ’s influence distribution

Original: I i ( u ti | l ti ) , which may become non-stationary.
Revised: I i ( u ti | l ti ) , which may become non-stationary.

Original: This renders previously computed influencesuseless because they no longer capture the trueresponse of the global system.
Revised: This renders previously computed influencesuseless because they no longer capture the trueresponse of the global system.

Original: This restriction, however, prevents IBA from unlocking its full potential.
Revised: This restriction, however, prevents IBA fromunlocking its full potential.

Original: The fact that eachagent’s IALS is independent of the oth‘ersmeans that the computations can be distributed among different processes that can run in parallel.
Revised: The fact that eachagent’s IALS is independent of the others meansthat the computations can be distributed amongdifferent processes that can run in parallel.

Original: Hence, putting aside the non-stationarity issue, and assuming no overhead costs in spawning anincreasing number of processes, the total runtime of the method would stay constant if the dimensionality of the global system grew, either because the number of non-local variables or the number ofagents increased.
Revised: Hence, putting aside the non-stationarity issue, andassuming no overhead costs in spawning an increasing number of processes, the total runtime ofthe method would stay constant if the dimensionality of the global system grew, either because thenumber of non-local variables or the number of agents increased.

Original: This is in contrast to having agents learn simultaneously in the same GS, in whichcase larger environments imply longer runtimes.
Revised: This is in contrast to having agentslearn simultaneously in the same GS, in which case larger environments imply longer runtimes.

Original: Moreover, since each IALS simulates only a portionof the environment the total amount of memory space needed would be split among the differentprocessors.
Revised: Moreover, since each IALS simulates only a portion of the environment the total amount of memoryspace needed would be split among the different processors.

Original: Hence, we could run the simulation on multiple machines with small memory rather thanone big machine with very large memory.
Revised: Hence, we could run the simulation onmultiple machines with small memory rather than one big machine with very large memory.

Original: In principle, one could prevent the AIPs from becoming stale by simply updating all { ˆ I θ i ( u ti | l ti ) } i ∈ Nevery time any of the other agents changes its policy.
Revised: In principle, one could prevent the AIPs from becoming stale by simply updating all { ˆ I θ i ( u ti | l ti ) } i ∈ Nevery time any of the other agents changes its policy.

Original: However, this creates a difficult moving targetproblem, and makes the whole method very inefficient since, especially in deep RL, policies areupdated very frequently.
Revised: However, this creates a difficult moving targetproblem, and makes the whole method very inefficient since, especially in deep RL, policies areupdated very frequently.

Original: Fortunately, as we argue in the following, in many cases, paying the extracost of retraining the AIPs is neither necessary nor desirable. 4.1.1 Multiple joint policies may induce the same influence distribution
Revised: Fortunately, as we argue in the following, in many cases, paying the extracost of retraining the AIPs is neither necessary nor desirable.4.1.1 Multiple joint policies may induce the same influence distribution

Original: In the following, we show that multiple joint policies may often map onto the same influencedistribution
Revised: In the following, we show that multiple joint policies may often map onto the same influencedistribution

Original: I i ( u ti | l ti ) ∈
Revised: I i ( u ti | l ti ) ∈

Original: Ψ i for agent i ∈ N .
Revised: Ψ i for agent i ∈ N .

Original: Lemma 1.
Revised: Lemma 1.

Original: Let Π = ×
Revised: Let Π = ×

Original: i ∈ N Π
Revised: i ∈ N Π

Original: i be the product space of joint policies with Π i being the set of policesfor agent i .
Revised: i be the product space of joint policies with Π i being the set of policesfor agent i .

Original: Moreover, let Ψ = × i ∈ N Ψ
Revised: Moreover, let Ψ = × i ∈ N Ψ

Original: i be the product space of joint influences, with Ψ i being theset of influence distributions for agent i .
Revised: i be the product space of joint influences, with Ψ i being theset of influence distributions for agent i .

Original: Every joint policy π ∈ Π induces exactly one influencedistribution
Revised: Every joint policy π ∈ Π induces exactly one influencedistribution

Original: I i ( u ti | l ti ) ∈
Revised: I i ( u ti | l ti ) ∈

Original: Ψ i for every agent i ∈ N .
Revised: Ψ i for every agent i ∈ N .

Original: Proposition 1.
Revised: Proposition 1.

Original: The space of joint policies Π = × i ∈ N Π i is necessarily greater than or equal to thespace of joint influences Ψ
Revised: The space of joint policies Π = × i ∈ N Π i is necessarily greater than or equal to thespace of joint influences Ψ

Original: = × i ∈ N Ψ
Revised: = × i ∈ N Ψ

Original: i , | Π | ≥ | Ψ | .
Revised: i , | Π | ≥ | Ψ | .

Original: Moreover, there exist local-form fPOSGs forwhich the inequality is strict.
Revised: Moreover, there exist local-form fPOSGs forwhich the inequality is strict.

Original: The advantages of this result were shown empirically by Witwicki and Durfee (2010), who demonstrated that planning times can be reduced by searching the space of joint influences rather than thespace of joint policies, which is often much larger.
Revised: The advantages of this result were shown empirically by Witwicki and Durfee (2010), who demonstrated that planning times can be reduced by searching the space of joint influences rather than thespace of joint policies, which is often much larger.

Original: In fact, in the extreme case of local transitionindependence (Becker et al., 2003), 2 we have that for all joint policies π there is a single { I i } i ∈ N
Revised: In fact, in the extreme case of local transitionindependence (Becker et al., 2003), 1 we have that for all joint policies π there is a single { I i } i ∈ N

Original: Corollary 1.
Revised: Corollary 1.

Original: Let agent
Revised: Let agent

Original: i ’s influence sources u ti be independent of the other agents’ actions a − i .
Revised: i ’s influence sources u ti be independent of the other agents’ actions a − i .

Original: Then, for any joint policy π ∈ Π , there is a unique influence distribution I ∗ i
Revised: Then, for any joint policy π ∈ Π , there is a unique influence distribution I ∗ i

Original: ∈ Ψ i for every agenti ∈ N and | Π | ≫ | Ψ | = 1 .
Revised: ∈ Ψ i for every agenti ∈ N and | Π | ≫ | Ψ | = 1 .

Original: The result above implies that, in this particular case, we would only need to train the AIPs once at thebeginning.
Revised: The result above implies that, in this particular case, we would only need to train the AIPs once at thebeginning.

Original: Although we do not expect the situation in Corollary 1 to be the norm, we do believe thatin many scenarios, such as in the two environments we explore here, we would not need to retrain the
Revised: Although we do not expect the situation in Corollary 1 to be the norm, we do believe thatin many scenarios, such as in the two environments we explore here, we would not need to retrain the

Original: AIPs very often because similar joint policies will influence the local regions in very similar, if not inthe same, ways.
Revised: AIPs very often because similar joint policies will influence the local regions in very similar, if not inthe same, ways.

Original: Furthermore, the next result shows that even when this is not the case, an outdated I icomputed from an old joint policy might still produce the same optimal policy for agent i .
Revised: Furthermore, the next result shows that even when this is not the case, an outdated I icomputed from an old joint policy might still produce the same optimal policy for agent i .

Original: 4.1.2 Multiple influence distributions may induce the same optimal policy
Revised: 4.1.2 Multiple influence distributions may induce the same optimal policy

Original: We use the simulation lemma (Kearns and Singh, 2002) to prove that if two influence distributionsare similar enough they will induce the same optimal policy.
Revised: We use the simulation lemma (Kearns and Singh, 2002) to prove that if two influence distributionsare similar enough they will induce the same optimal policy.

Original: Lemma 2.
Revised: Lemma 2.

Original: Let M 

Revised: Let M

Original: i
Revised: i

Original: and M 2 i be two IALMS differing only on their influence distributions
Revised: and M 2 i be two IALMS differing only on their influence distributions

Original: I 1 i ( u ti | l ti )and I 

Revised: I 1 i ( u ti | l ti )and I

Original: i ( u ti | l ti ) .
Revised: i ( u ti | l ti ) .

Original: Let Q π i M 1 i and Q π
Revised: Let Q π i M 1 i and Q π

Original: i M 

Revised: i M

Original: i be the value functions induced by M 1 i and M 2 i for the same π i .
Revised: i be the value functions induced by M 1 i and M 2 i for the same π i .

Original: If
Revised: If

Original: I 1 i and I [-
-] i satisfy
Revised: I 1 i and I [-
-]2 i satisfy(cid:88)

Original: [Equation]
Revised: [Equation]

Original: ξ
Revised: ξ

Original: [Equation]
Revised: [Equation]

Original: for all π i , h ti , and a ti , where H is the horizon and ¯ R =
Revised: for all π i , h ti , and a ti , where H is the horizon and ¯ R =

Original: ||
Revised: ||

Original: R || ∞
Revised: R || ∞

Original: Intuitively, Lemma 2 shows that the difference in value between M 1 i an M 2 i is upper-bounded by themaximum difference between I 1 i and I 2 i times a constant.
Revised: Intuitively, Lemma 2 shows that the difference in value between M 1 i an M 2 i is upper-bounded by themaximum difference between I 1 i and I 2 i times a constant.

Original: Actually, if the action-gap (Farahmand,2011) (i.e. value difference between the best and the second best action) in one of the IALMs is largerthan twice the difference between the Q π i M 1 and Q π i M 2 the IALMs share the same optimal policy.
Revised: Actually, if the action-gap (Farahmand,2011) (i.e. value difference between the best and the second best action) in one of the IALMs is largerthan twice the difference between the Q π i M 1 and Q π i M 2 the IALMs share the same optimal policy.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let M 1 i and M 2 i be two IALMS differing only on their influence distributions
Revised: Let M 1 i and M 2 i be two IALMS differing only on their influence distributions

Original: I 1 i ( u ti | l ti )and I 

Revised: I 1 i ( u ti | l ti )and I

Original: i ( u ti | l ti ) .
Revised: i ( u ti | l ti ) .

Original: M 1 i and M 2 i induce the same optimal policy π ∗ if, for some ∆ ,
Revised: M 1 i and M 2 i induce the same optimal policy π ∗ if, for some ∆ ,

Original: [Equation]
Revised: [Equation]

Original: h ti , ¯ a ti ) − Q π ∗
Revised: h ti , ¯ a ti ) − Q π ∗

Original: i M 

Revised: i M

Original: i ( h ti , ˆ a ti ) > 2∆ ∀ h ti , ˆ a ti ̸
Revised: i ( h ti , ˆ a ti ) > 2∆ ∀ h ti , ˆ a ti ̸

Original: = ¯ a ti with (cid:12)(cid:12)(cid:12) Q π i M 

Revised: = ¯ a ti with (cid:12)(cid:12)(cid:12) Q π i M

Original: i
Revised: i

Original: ( h ti , a ti ) − Q π
Revised: ( h ti , a ti ) − Q π

Original: i M 

Revised: i M

Original: i ( h ti , a ti ) (cid:12)(cid:12)(cid:12) ≤ ∆ ∀ h ti , a ti , π i , (3) where ¯ a ti = arg max a ti Q π ∗ i M 

Revised: i ( h ti , a ti ) (cid:12)(cid:12)(cid:12) ≤ ∆ ∀ h ti , a ti , π i , (3) where ¯ a ti = arg max a ti Q π ∗ i M

Original: i
Revised: i

Original: ( h ti , a ti )
Revised: ( h ti , a ti )

Original: Combining Lemma 2 and Theorem 1, we see that, because the difference in value between M 1 i an
Revised: Hence, combining Lemma 2 and Theorem 1, ∆ can be chosen as ¯ R

Revised: ( H − t )

Original: M 2 i depends on ξ (Lemma 2), the closer the distributions I 1 i and I 2 i are, the more likely it is that M 1 iand M 2 i share the same optimal policy.
Revised: ( H − t +1)ξ , which means thecloser the distributions I 1 i and I 2 i are, the more likely they are to elicit the same optimal policy.

Original: 4.2 Algorithm
Revised: 4.2 Algorithm

Original: After the analysis above, we are now ready to present our method, which we call Distributed InfluenceAugmented Local Simulators (DIALS).
Revised: After the analysis above, we are now ready to present our method, which we call Distributed InfluenceAugmented Local Simulators (DIALS).

Original: Algorithm 1 describes how we can train multi-agent systemswith DIALS.
Revised: Algorithm 1 describes how we can train multi-agent systemswith DIALS.

Original: As mentioned earlier, the key advantage in using DIALS is that simulations can bedistributed among different processes, and thus training can be fully parallelized.
Revised: As mentioned earlier, the key advantage in using DIALS is that simulations can bedistributed among different processes, and thus training can be fully parallelized.

Original: This enables MARLto scale to very large systems with many learning agents.
Revised: This enables MARLto scale to very large systems with many learning agents.

Original: Moreover, following from our theoreticalresults, AIP training, which can also be done in parallel, is performed only every certain number oftimesteps.
Revised: Moreover, following from our theoreticalresults, AIP training, which can also be done in parallel, is performed only every certain number oftimesteps.

Original: The hyperparameter F in Algorithm 1 controls the AIPs’ training frequency.Algorithm 1: repeat 2:
Revised: The hyperparameter F in Algorithm 1 controls the AIPs’ training frequency.

Original: Collect datasets { D i } i ∈ N from GS ▷ See Algorithm2 in Appendix D 3: in parallel, for i ∈ N do 4: Train AIP ˆ I θ i on dataset D i ▷ See Section 3.2 5: end for 6: in parallel, for i ∈ N do 7: for do ▷ F
Revised: Algorithm

Original: AIPs’ training frequency 8:

Original: Simulate trajectories with IALS ⟨ ˙ T i , ˙ R i , ˙ O i , ˆ I θ i ⟩ ▷ See Algorithm 3 in Appendix D 9:
Revised: O i , ˆ I θ i ⟩ ▷ See Algorithm 3 in Appendix C 10:

Original: Train policy π i ▷
Revised: Train policy π i ▷

Original: Using any standard RL method10: end for 11: end for 12:  until end of training
Revised: Using any standard RL method 11: end for 12: end for 13: until end of training

Original: 4.3 Mitigating the negative effects of simultaneous learning
Revised: 4.3 Mitigating the negative effects of simultaneous learning

Original: The results in the previous section showed that the AIPs may not need to be retrained every singletime policies are updated, as the influence distributions may often stay the same or vary only a little.
Revised: The results in the previous section showed that the AIPs may not need to be retrained every singletime policies are updated, as the influence distributions may often stay the same or vary only a little.

Original: Yet, we now argue that, even when changes in the joint policy do affect the influence distributions
Revised: Yet, we now argue that, even when changes in the joint policy do affect the influence distributions

Original: I i ( u ti | l ti ) significantly, it may be advantageous not to retrain the AIPs.
Revised: I i ( u ti | l ti ) significantly, it may be advantageous not to retrain the AIPs.

Original: First, we have already mentioned that when all agents learn simultaneously in the same simulator thetransition dynamics often look non-stationary from the perspective of each individual agent.
Revised: First, we have already mentioned that when all agents learn simultaneously in the same simulator thetransition dynamics often look non-stationary from the perspective of each individual agent.

Original: This mayresult in sudden performance drops caused by oscillations in the value targets (Claus and Boutilier,1998).
Revised: This mayresult in sudden performance drops caused by oscillations in the value targets (Claus and Boutilier,1998).

Original: In contrast, when using independent IALS to train our agents, the transition dynamics remainstationary unless we update the AIPs.
Revised: In contrast, when using independent IALS to train our agents, the transition dynamics remainstationary unless we update the AIPs.

Original: Hence, by not updating the AIPs too frequently, we get a biasedbut otherwise more consistent learning signal that the agents can rely on to improve their policies.
Revised: Hence, by not updating the AIPs too frequently, we get a biasedbut otherwise more consistent learning signal that the agents can rely on to improve their policies.

Original: Second, we posit that the poor empirical convergence of manyoff-the-shelf Deep RL methods(Hernandez-Leal et al., 2017; Yu et al., 2021) is also because stochastic gradient descent updatesoften result in policies that perform worse than the previous ones.
Revised: Second, we posit that the poor empirical convergence of many Deep MARL methods is also becausestochastic gradient descent updates often result in policies that perform worse than the previous ones.

Original: Thus, when learning together,agents may try to adapt to other agents’ poor performing polices.
Revised: Thus, when learning together, agents may try to adapt to other agents’ poor performing polices.

Original: These policies, however, are likelyto be temporary as they are just a result of the inherent stochasticity of the learning process.
Revised: Thesepolicies, however, are likely to be temporary as they are just a result of the inherent stochasticity of thelearning process.

Original: Similarly,in many environments, agents shall take exploratory actions before they can improve their policies,which may also negatively impact cooperation if they learn simultaneously (Zhang et al., 2009, 2010).
Revised: Similarly, in many environments, agents shall take exploratory actions before theycan improve there policies, which may also negatively impact cooperation if they learn simultaneously(Zhang et al., 2009, 2010).

Original: In our case, we can again benefit from the fact that the AIPs need to be purposely retrained, and do soonly when the policies of the other agents have improved sufficiently.
Revised: In our case, we can again benefit from the fact that the AIPs need to bepurposely retrained, and do so only when the policies of the other agents have improved sufficiently.

Original: Even though, further theoretical analysis would be needed to be more conclusive about the benefitsof using independent simulators, the observations above give reasons to believe that what we initiallydescribed as a problem may in fact be an advantage.
Revised: Even though, further theoretical analysis would be needed to be more conclusive about the benefitsof using independent simulators, the observations above give reasons to believe that what we initiallydescribed as a problem may in fact be an advantage.

Original: This view is also supported by our experiments.
Revised: This view is also supported by our experiments.

Original:  Experiments
Revised:  Experiments

Original: The goal of the experiments is to: (1) test whether we can reduce training times by replacing GS with
Revised: The goal of the experiments is to: (1) test whether we can reduce training times by replacing GS with

Original: DIALS, (2) investigate how the method scales to large environments with many learning agents, (3)evaluate the convergence benefits of using separate simulators to train agents rather than a single GS,and (4) study the effect of the AIPs’ training frequency F on the agents’ learning performance.
Revised: DIALS, (2) investigate how the method scales to large environments with many learning agents, (3)evaluate the convergence benefits of using separate simulators to train agents rather than a single GS,and (4) study the effect of the AIPs’ training frequency F on the agents’ learning performance.

Original: 5.1 Experimental setup
Revised: 5.1 Experimental setup

Original: Agents are trained independently with PPO (Schulman et al., 2017) 3 on (1) the global simulator(GS), (2) distributed influence-augmented local simulators (DIALS) with AIPs trained periodicallyon datasets collected from the GS using the most recent joint policy, (3) DIALS with untrained
Revised: Agents are trained independently with PPO (Schulman et al., 2017) 2 on (1) the global simulator(GS), (2) distributed influence-augmented local simulators (DIALS) with AIPs trained periodicallyon datasets collected from the GS using the most recent joint policy, (3) DIALS with untrained

Original: AIPs (untrained-DIALS).
Revised: AIPs (untrained-DIALS).

Original: To measure the agent’s performance, training is interleaved with periodicevaluations on the GS.
Revised: To measure the agent’s performance, training is interleaved with periodicevaluations on the GS.

Original: The results are averaged over 10 random seeds on all except on the largestscenarios ( 10 × 10 ) for which, due to computational limitations we could only run 5 seeds.
Revised: The results are averaged over 10 random seeds on all except on the largestscenarios ( 10 × 10 ) for which, due to computational limitations we could only run 5 seeds.

Original: We reportthe mean return of all learning agents.
Revised: We reportthe mean return of all learning agents.

Original: We also compare the simulators in terms of total runtime.
Revised: We also compare the simulators in terms of total runtime.

Original: For DIALS this includes: the agents’ training time, the AIPs’ training time, and the time for datacollection.
Revised: For DIALS this includes: the agents’ training time, the AIPs’ training time, and the time for datacollection.

Original: 5.2 Environments
Revised: 5.2 Environments

Original: Traffic control The first domain we consider is a multi-agent variant of the traffic control benchmarkproposed by Vinitsky et al.
Revised: Traffic control The first domain we consider is a multi-agent variant of the traffic control benchmarkproposed by Vinitsky et al.

Original: In this scenario agents are requested to manage the lights of abig traffic network.
Revised: In this scenario agents are requested to manage the lights of abig traffic network.

Original: Each agent controls a single traffic light and can only observe cars when theyare inside the intersection’s local neighborhood.
Revised: Each agent controls a single traffic light and can only observe cars when theyare inside the intersection’s local neighborhood.

Original: Their goal is to maximize the average speed of carswithin their respective intersections.
Revised: Their goal is to maximize the average speed of carswithin their respective intersections.

Original: To demonstrate the scalability of the method we evaluate DIALSon four different variants of the traffic network with 4 , 25 , 49 , and even 100 intersections (agents).
Revised: To demonstrate the scalability of the method we evaluate DIALSon four different variants of the traffic network with 4 , 25 , 49 , and even 100 intersections (agents).

Original: A screenshot of the traffic network with 25 intersections is shown in Appendix G. The GS and LSare built using Flow (MIT License)
Revised: A screenshot of the traffic network with 25 intersections is shown in Appendix E. The GS and LSare built using Flow (MIT License)

Original: (Wu et al., 2017) and SUMO (Eclipse Public License Version 2)(Lopez et al., 2018).
Revised: (Wu et al., 2017) and SUMO (Eclipse Public License Version 2)(Lopez et al., 2018).

Original: The GS simulates the entire traffic network while each LS models only the localneighborhood of each intersection i ∈ N (Figure 10 in Appendix G).
Revised: The GS simulates the entire traffic network while each LS models only the localneighborhood of each intersection i ∈ N (Figure 6 in Appendix E).

Original: We use the same LS for everyagent-intersection but since, depending on where they are located, they are influenced differentlyby the rest of the traffic network, we have separate AIPs, { I θ i } i ∈ N , for each them.
Revised: We use the same LS for everyagent-intersection but since, depending on where they are located, they are influenced differentlyby the rest of the traffic network, we have separate AIPs, { I θ i } i ∈ N , for each them.

Original: These arefeedforward neural networks with the same architecture but different weights θ i , trained periodicallywith frequency F on datasets { D i } i ∈ N collected from the GS.
Revised: These arefeedforward neural networks with the same architecture but different weights θ i , trained periodicallywith frequency F on datasets { D i } i ∈ N collected from the GS.

Original: The influence sources u
Revised: The influence sources u

Original: it are binaryvariables indicating whether or not a car will be entering from each the four incoming lanes.
Revised: it are binaryvariables indicating whether or not a car will be entering from each the four incoming lanes.

Original: 4 Timesteps 1e
Revised: 4 Timesteps 1e

Original: A v e r a g e R e t u r n(1a)
Revised: A v e r a g e R e t u r n

Original: GSuntrained-DIALSDIALS F=

Revised: GSuntrained-DIALSDIALS F=

Original: Mhand-coded 4 25 49 

Revised: M 4 25 49

Original: A v e r a g
Revised: A v e r a g

Original: e R e t u r n(2a) 4 25 

Revised: e R e t u r n 4 25

Original: T o t a l r un t
Revised: T o t a l r un t

Original: i m e ( h )(3a)
Revised: i m e ( h )

Original: Number of agents (a) Traffic    4 Timesteps 1e

Revised: Number of agents (a) Traffic    4 Timesteps 1e

Original: A v e r a g e R e t u r n(1b)
Revised: A v e r a g e R e t u r n

Original: GSuntrained-DIALSDIALS F=

Revised: GSuntrained-DIALSDIALS F=

Original: Mhand-coded 4 25 49 

Revised: M 4 25 49

Original: A v e r a g
Revised: A v e r a g

Original: e R e t u r n(2b) 4 25 

Revised: e R e t u r n 4 25

Original: T o t a l r un t
Revised: T o t a l r un t

Original: i m e ( h )(3b)
Revised: i m e ( h )

Original: Number of agents (b) Warehouse
Revised: Number of agents (b) Warehouse

Original: Warehouse
Revised: Warehouse

Original: Commissioning The second domain we consider is a warehouse commissioning task(Suau et al., 2022).
Revised: Commissioning The second domain we consider is a warehouse commissioning task(Suau et al., 2022).

Original: A team of robots (blue) need to fetch the items (yellow) that appear withprobability 0 .
Revised: A team of robots (blue) need to fetch the items (yellow) that appear withprobability 0 .

Original: 02 on the shelves (dashed black lines) of the warehouse (see Figure 9 in Appendix G).
Revised: 02 on the shelves (dashed black lines) of the warehouse (see Figure 5 in Appendix E).

Original: Each robot has been designated a 5 × 5 square region and can only collect the items that appear onthe shelves at the edges.
Revised: Each robot has been designated a 5 × 5 square region and can only collect the items that appear onthe shelves at the edges.

Original: The regions overlap so that each of the 4 item shelves in a robot’s region isshared with one of its 4 neighbors.
Revised: The regions overlap so that each of the 4 item shelves in a robot’s region isshared with one of its 4 neighbors.

Original: The robots receive as observations a bitmap encoding their ownlocation and a set of 12 binary variables that indicate whether or not a given item needs to be collected.
Revised: The robots receive as observations a bitmap encoding their ownlocation and a set of 12 binary variables that indicate whether or not a given item needs to be collected.

Original: The robots, however, cannot see the location of the other robots even though all of them are directlyor indirectly influencing each other through their actions.
Revised: The robots, however, cannot see the location of the other robots even though all of them are directlyor indirectly influencing each other through their actions.

Original: We built four variants of the warehousewith 4 , 25 , 49 , and 100 robots (agents).
Revised: We built four variants of the warehousewith 4 , 25 , 49 , and 100 robots (agents).

Original: A screenshot of the warehouse with 25 robots is shown in
Revised: A screenshot of the warehouse with 25 robots is shownin

Original: Appendix G. The GS simulates the entire warehouse while the LS models only a 5 × 5 square region(Figure 10 in Appendix G).
Revised: Appendix E. The GS simulates the entire warehouse while the LS models only a 5 × 5 squareregion (Figure 6 in Appendix E).

Original: We use the same LS for every robot (agent) but since, depending onwhere they are located they are influenced differently by the rest of the robots, we have separate
Revised: We use the same LS for every robot (agent) but since, dependingon where they are located they are influenced differently by the rest of the robots, we have separate

Original: AIPs, { ˆ I θ i } i ∈ N , for each of them.
Revised: AIPs, { ˆ I θ i } i ∈ N , for each of them.

Original: These are GRUs (Cho et al., 2014) with the same architecture butdifferent weights θ i , which we train periodically with frequency F on datasets { D i } i ∈ N collectedfrom the GS.
Revised: These are GRUs (Cho et al., 2014) with the same architecture butdifferent weights θ i , which we train periodically with frequency F on datasets { D i } i ∈ N collectedfrom the GS.

Original: Robot i ’s influence sources u ti encode the location of the four neighbor robots.
Revised: Robot i ’s influence sources u ti encode the location of the four neighbor robots.

Original: If its
Revised: If its

Original: AIP ˆ I θ i predicts that any of the neighbor robots is at one of the 12 cells within its region, and there isan active item on that cell, that item is removed and robot i can no longer collect it.
Revised: AIP ˆ I θ i predicts that any of the neighbor robots is at one of the 12 cells within its region, and there isan active item on that cell, that item is removed and robot i can no longer collect it.

Original: 5.3 Results
Revised: 5.3 Results

Original: GS vs. DIALS
Revised: GS vs. DIALS

Original: The two plots on the left of Figures 3a and 3b show the average return as a functionof the number of timesteps obtained with GS, DIALS, and untrained-DIALS on the 4 -agent trafficand warehouse environments.
Revised: The two plots on the left of Figures 3a and 3b show the average return as a functionof the number of timesteps obtained with GS, DIALS, and untrained-DIALS on the 4 -agent trafficand warehouse environments.

Original: Shaded areas indicate the standard error of the mean.
Revised: Shaded areas indicate the standard error of the mean.

Original: Agents aretrained for 4M timesteps on all three simulators.
Revised: Agents aretrained for 4M timesteps on all three simulators.

Original: The results reveal that, while agents trained on
Revised: The results reveal that, while agents trained on

Original: DIALS seem to converge steadily towards similar high-performing policies, agents trained with the
Revised: DIALS seem to converge steadily towards similar high-performing policies, agents trained with the

Original: GS often get stuck in local minima, hence the poor mean episodic reward and large standard errorobtained with the GS relative to that of the DIALS.
Revised: GS often get stuck in local minima, hence the poor mean episodic reward and large standard errorobtained with the GS relative to that of the DIALS.

Original: In contrast, the low performance of agents trainedwith the untrained-DIALS indicates that estimating the influences correctly is important for learninggood policies.
Revised: In contrast, the low performance of agents trainedwith the untrained-DIALS indicates that estimating the influences correctly is important for learninggood policies.

Original: It is worth noting that the gap between the GS and the DIALS is larger in Figure 3bthan in Figure 3a.
Revised: It is worth noting that the gap between the GS and the DIALS is larger in Figure 3bthan in Figure 3a.

Original: We posit that this is because, in the warehouse domain, agents are more stronglycoupled.
Revised: We posit that this is because, in the warehouse domain, agents are more stronglycoupled.

Original: The learning curves for the other scenarios are provided in Appendix E together with furtherdiscussion on these results.
Revised: The learning curves for the other scenarios are provided in Appendix D together with furtherdiscussion on these results.

Original: Scalability The benefits of parallelization are more apparent when moving to larger environments.
Revised: Scalability The benefits of parallelization are more apparent when moving to larger environments.

Original: The two bar plots in Figures 3a and 3b depict the final average return and the total run time of training4 , 25 , 49 and 100 agents on the two tasks for 4 M timesteps.
Revised: The two bar plots in Figures 3a and 3b depict the final average return and the total run time of training4 , 25 , 49 and 100 agents on the two tasks for 4 M timesteps.

Original: The plots show that DIALS scalesfar better than the GS to larger problem sizes, while also yielding better performing policies.
Revised: The plots show that DIALS scalesfar better than the GS to larger problem sizes, while also yielding better performing policies.

Original: Forexample, training 100 agents on the traffic network takes less than 6 hours with the DIALS, whereastraining them with the GS would take more than 10 days .
Revised: Forexample, training 100 agents on the traffic network takes less than 6 hours with the DIALS, whereastraining them with the GS would take more than 10 days .

Original: This is a speedup factor of 40 .
Revised: This is a speedup factor of 40 .

Original: In fact,since the maximum execution time allowed by our computer cluster is 1 week, the results reportedfor the GS in the scenarios of size 10 × 10 do not correspond to 4M timesteps but the equivalent of 1week of training. We would also like to point out that, disregarding the overhead costs associated
Revised: In fact,since the maximum execution time allowed by our computer cluster is 1 week, the results reported

Original: [Figure]
Revised: [Figure]

Original: a
Revised: for the GS in the scenarios of size 10 × 10 do not correspond to 4M timesteps but the equivalent of 1week of training.

Original: [Figure] [Figure] with multiprocessing, the DIALS runtime should remain constant independently of the problemsize.
Revised: We would also like to point out that, disregarding the overhead costs associatedwith multiprocessing, the DIALS runtime should remain constant independently of the problemsize.

Original: However, to update the AIPs, new samples are collected form the GS, which does increasethe runtime.
Revised: However, to update the AIPs, new samples are collected form the GS, which does increasethe runtime.

Original: This explains the gap between DIALS and untrained-DIALS.
Revised: This explains the gap between DIALS and untrained-DIALS.

Original: That said, the number ofsamples needed to update the AIPs (80K for traffic and 10K for warehouse) is significantly lowerthan the samples needed to train the agents (4M), which is why the runtime difference between GSand DIALS is so large.
Revised: That said, the number ofsamples needed to update the AIPs (80K for traffic and 10K for warehouse) is significantly lowerthan the samples needed to train the agents (4M), which is why the runtime difference between GSand DIALS is so large.

Original: A table with a breakdown of the runtimes is given in Appendix H.
Revised: A table with a breakdown of the runtimes is given in Appendix F.

Original: AIPs’ training frequency Our first results have already demonstrated that isolating the agentsin separate simulators and not updating the AIPs too frequently can be beneficial for convergence.
Revised: AIPs’ training frequency Our first results have already demonstrated that isolating the agentsin separate simulators and not updating the AIPs too frequently can be beneficial for convergence.

Original: We now further investigate this phenomenon by evaluating the agents’ learning performance fordifferent values of the hyperparameter F .
Revised: We now further investigate this phenomenon by evaluating the agents’ learning performance fordifferent values of the hyperparameter F .

Original: The two plots on the left of Figures 4a and 4b show thelearning curves for agents trained on DIALS where F is set to 100 K, 500 K, 1 M, and 4 M timesteps.
Revised: The two plots on the left of Figures 4a and 4b show thelearning curves for agents trained on DIALS where F is set to 100 K, 500 K, 1 M, and 4 M timesteps.

Original: In the traffic domain, the gap between the green and the purple curve (Figure 4a) suggests that it isimportant to retrain the AIPs at least every 1 M timesteps, such that agents become aware of changesin the other agents’ policies.
Revised: In the traffic domain, the gap between the green and the purple curve (Figure 4a) suggests that it isimportant to retrain the AIPs at least every 1 M timesteps, such that agents become aware of changesin the other agents’ policies.

Original: In contrast, in the warehouse domain (Figure 4b), we see that trainingthe AIPs only once at the beginning (DIALS F = 4 M) seems sufficient.
Revised: In contrast, in the warehouse domain (Figure 4b), we see that trainingthe AIPs only once at the beginning (DIALS F = 4 M) seems sufficient.

Original: In fact, updating the AIPstoo frequently (DIALS F = 100 K) is detrimental to the agents’ performance.
Revised: In fact, updating the AIPstoo frequently (DIALS F = 100 K) is detrimental to the agents’ performance.

Original: This is consistent withour hypothesis in Section 4.3.
Revised: This is consistent withour hypothesis in Section 4.3.

Original: The plots on the right show the average cross-entropy (CE) loss of the
Revised: The plots on the right show the average cross-entropy (CE) loss of the

Original: AIPs evaluated on trajectories sampled from the GS.
Revised: AIPs evaluated on trajectories sampled from the GS.

Original: As explained in Section 4 since all agents learnsimultaneously, the influence distributions { I ( u ti | l ti ) } i ∈ N are non-stationary.
Revised: As explained in Section 4 since all agents learnsimultaneously, the influence distributions { I ( u ti | l ti ) } i ∈ N are non-stationary.

Original: For this reason, we seethat the CE loss changes as the policies of the other agents are updated.
Revised: For this reason, we seethat the CE loss changes as the policies of the other agents are updated.

Original: We can also see how the CEloss decreases when the AIPs are retrained, which happens more or less frequently depending onthe hyperparameter F .
Revised: We can also see how the CEloss decreases when the AIPs are retrained, which happens more or less frequently depending onthe hyperparameter F .

Original: Note that the CE not only measures the distance between the two probabilitydistributions but also the absolute entropy.
Revised: Note that the CE not only measures the distance between the two probabilitydistributions but also the absolute entropy.

Original: In the warehouse domain, the neighbor robots’ locationsbecome more predictable (lower entropy) as their policies improve.
Revised: In the warehouse domain, the neighbor robots’ locationsbecome more predictable (lower entropy) as their policies improve.

Original: This explains why in the firstplot from the right the CE loss decreases even though the AIPs are not updated.
Revised: This explains why in the firstplot from the right the CE loss decreases even though the AIPs are not updated.

Original: Also in the sameplot, even though by the end of training DIALS F = 4 M is highly inaccurate, as evidenced by thegap between the purple and the other curves, it is still good enough to train policies that match theperformance of those trained with DIALS F = 500 K and F = 

Revised: Also in the sameplot, even though by the end of training DIALS F = 4 M is highly inaccurate, as evidenced by thegap between the purple and the other curves, it is still good enough to train policies that match theperformance of those trained with DIALS F = 500 K and F =

Original: M. This is in line with our results in
Revised: M. This is in line with our results in

Original: Section 4.
Revised: Section 4.

Original: The same plots for the rest of scenarios are provided in Appendix E.
Revised: The same plots for the rest of scenarios are provided in Appendix D.

Original:  Conclusion
Revised:  Conclusion

Original: This paper has offered a practical solution that allows training large networked systems with manyagents in just a few hours.
Revised: This paper has offered a practical solution that allows training large networked systems with manyagents in just a few hours.

Original: We showed how to decompose these systems into multiple sub-regions suchthat we could build distributed influence-augmented local simulators (DIALS).
Revised: We showed how to decompose these systems into multiple sub-regions suchthat we could build distributed influence-augmented local simulators (DIALS).

Original: The key advantageof DIALS is that simulations can be distributed among different processes, and thus training can befully parallelized.
Revised: The key advantageof DIALS is that simulations can be distributed among different processes, and thus training can befully parallelized.

Original: To account for the interactions between the different sub-regions, the simulatorsare equipped with approximate influence predictors (AIPs), which are trained periodically on realtrajectories sampled from a global simulator (GS).
Revised: To account for the interactions between the different sub-regions, the simulatorsare equipped with approximate influence predictors (AIPs), which are trained periodically on realtrajectories sampled from a global simulator (GS).

Original: We demonstrated that, although with DIALSagents learn simultaneously, training the AIPs very frequently is neither necessary nor desirable.
Revised: We demonstrated that, although with DIALSagents learn simultaneously, training the AIPs very frequently is neither necessary nor desirable.

Original: Ourresults reveal that DIALS not only enables MARL to scale up but also mitigates the non-stationarityissues of simultaneous learning.
Revised: Ourresults reveal that DIALS not only enables MARL to scale up but also mitigates the non-stationarityissues of simultaneous learning.

Original: Future work could analyse this phenomenon from a theoreticalperspective, or studystudy how to adapt this approach to more strongly coupled domains wherefrequent training of the AIPs is important.
Revised: Future work could analyse this phenomenon from a theoreticalperspective, or study how to retrain the AIPs without having to run the GS to generate new samples.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes](b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes] All proofs are includedin Appendix A.3.
Revised: [Yes] All proofs are included in Appendix A.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main ex-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main ex-

Original: perimental results (either in the supplemental material or as a URL)?
Revised: perimental results (either in the supplemental material or as a URL)?

Original: [Yes] Code
Revised: [Yes] Code

Original: and instructions on how to reproduce all the experimental results are included in the
Revised: and instructions on how to reproduce all the experimental results are included in the

Original: supplemental material.
Revised: supplemental material.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Section 5.1 for description of the experimental setup and
Revised: [Yes] See Section 5.1 for description of the experimental setup and

Original: Appendix J for a full list of the hyperparameter configurations used for the experiments
Revised: Appendix G for a full list of the hyperparameter configurations used for the experiments

Original: along with an explanation on how these were chosen.
Revised: along with an explanation on how these were chosen.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)?

Original: [Yes] The shaded areas in the learning curves and the blackvertical lines on the bar plots in Section 5.3 and Appendix E show the standard error ofthe mean.(d)
Revised: [Yes] The shaded areas in the learning curves and the blackvertical lines on the bar plots in Section 5.3 and Appendix D show the standard error ofthe mean.(d)

Original: Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] The computer requirements and
Revised: [Yes] The computer requirements and

Original: specifications along with a table showing a breakdown of the total runtimes for all
Revised: specifications along with a table showing a breakdown of the total runtimes for all

Original: experiments are provided in Appendix H.
Revised: experiments are provided in Appendix F.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation? [N/A] A Proofs
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: Lemma 1.

Original: Let Π = ×

Original: i ∈ N Π

Original: i be the product space of joint policies with Π i being the set of policesfor agent i .

Original: Moreover, let Ψ = × i ∈ N Ψ

Original: i be the product space of joint influences, with Ψ i being theset of influence distributions for agent i .

Original: Every joint policy π ∈ Π induces exactly one influencedistribution

Original: I i ( u ti | l ti ) ∈

Original: Ψ i for every agent i ∈ N .

Original: Proof.

Original: We will prove it by contradiction.

Original: Let us assume there is a single joint policy π that inducestwo different influence distributions

Original: I 1 i and I 2 i on agent i .

Original: From the definition of influence (Section4.1; Oliehoek et al. 2021)

Original: we have

Original: [Equation]

Original: (cid:88)

Original: [Equation]

Original: and

Original: I 2 i ( u

Original: [Equation]

Original: = (cid:88)

Original: [Equation]

Original: [Equation]

Original: First we see that, because ⟨ x t − 1 i , u t − 1 i , y t − 1 i ⟩ fully determines the Markov state s t − 1 , the first term inthe summation can be computed from the environment’s transition function, and thus

Original: [Equation]

Original: where 1 ( u t , s t ) is an indicator function that determines if the state s t is feasible in the context of u t .

Original: Further, we know that

Original: [Equation]

Original: [Equation]

Original: where P 1 ( u t − 1 i y t − 1 i , h t − 1 − i | l t ) and P 2 ( u t − 1 i y t − 1 i , h t − 1 − i | l t ) can be computed recursively as

Original: P

Original: [Equation]

Original: and

Original: [Equation]

Original: i ,o t − 1 − i

Original: O ( o t − 1 − i | x t − 1 i , u t − 1 i , y t − 1 i ) π − i ( a t − 2 − i | h t − 2 − i )

Original: P 2 ( u t − 1 i , y t − 1 i , h t − 2 − i | l ti ) , (10) with h t − 1 − i =

Original: ⟨ h − i,t − 2 , a − i,t − 2 , o t − 1 − i ⟩ .

Original: Then, if we further unroll equations (9) and (10) up to timestep0 , we see that all probability distributions in both cases are equivalent and we reach a contradiction.

Original: Hence,

Original: [Equation]

Original: Proposition 1.

Original: The space of joint policies Π = × i ∈ N Π i is necessarily greater than or equal to thespace of joint influences Ψ

Original: = × i ∈ N Ψ

Original: i , | Π | ≥ | Ψ | .

Original: Moreover, there exist local-form fPOSGs forwhich the inequality is strict.

Original: Proof.

Original: From Proposition 1 it follows that the space of joint influences Ψ is at most as large as thespace of joint policies Π , | Ψ | | > | Π | .

Original: Hence, we just need to show that in some cases Π is strictlygreater than Ψ , | Π | > | Ψ | .

Original: A clear example is that where each agent’s local region X

Original: i is independent of the other agents’ policiesπ − i (Becker et al., 2003).

Original: That is, the actions of other agents a − i have no effect on agent i ’s localstate transitions.

Original: From the definition of IALM (Definition 3) we know that, in our setting, a − i canonly affect the local state transitions through u i .

Original: Therefore, for the local transitions to be independentthe following should hold

Original: [Equation]

Original: The equation above reflects that only agent i can affect u ti .

Original: Thus, in the event of local transitionindependence, we have that

Original: [Equation]

Original: That is, for any joint policy π ∈ Π there is a unique influence distribution I ∗ i

Original: ∈ Ψ i for every agenti ∈ N , and thus, in this particular case, | Π | ≫ | Ψ | = 1 .

Original: To prove Lemma 2 we will use the following lemma.

Original: Lemma 3.

Original: [-Let I 
-]
Revised: [-Let I 
-]

Original: [-i ( u ti | l ti ) and I 
-]
Revised: [-i ( u ti | l ti ) and I 
-]

Original: i ( u ti | l ti ) be two different influence distributions with M 1 i and M 2 i beingthe IALMs induced by each of them respectively.

Original: Moreover, let P 1 ( h t +1 i | h ti , a ti ) and P 2 ( h t

Original: +1 i | h ti , a ti )denote the resulting local AOH transitions for M 1 i and M 2 i respectively.

Original: The following inequalityholds

Original: [Equation]

Original: ( h t +1 i | h ti , a ti ) (cid:12)(cid:12) ≤ (cid:88) l ti ,u ti

Original: P ( l ti | h ti ) (cid:12)(cid:12) I 1 ( u ti | l ti ) − I 2 ( u ti | l ti ) (cid:12)(cid:12)

Original: ∀

Original: h ti , a ti (14)

Original: Proof.

Original: (cid:88) h t +1 i (cid:12)(cid:12) P 1 ( h t

Original: +1 i | h ti , a ti ) − P 2 ( h t +1 i | h ti , a ti ) (cid:12)(cid:12) =

Original: (cid:88)

Original: o t +1 i (cid:12)(cid:12)(cid:12) (cid:88)

Original: x

Original: t +1 i

Original: O i ( o t +1 i | x t +1 i ) (cid:88)

Original: u ti ˙ T i ( x t

Original: +1 i | x ti , u ti , a ti ) (cid:88) l ti

Original: I 1 ( u ti | l ti ) P ( l ti | h ti ) − (cid:88) x

Original: t +1 i

Original: O i ( o t +1 i | x t +1 i ) (cid:88)

Original: u ti ˙ T i ( x t

Original: +1 i | x ti , u ti , a ti ) (cid:88) l ti

Original: I 2 ( u ti | l ti ) P ( l ti | h ti ) (cid:12)(cid:12)(cid:12) = (cid:88)

Original: o t +1 i (cid:12)(cid:12)(cid:12) (cid:88)

Original: x

Original: t +1 i

Original: O i ( o t +1 i | x t +1 i ) (cid:88)

Original: u ti ˙ T i ( x t

Original: +1 i | x ti , u ti , a ti ) (cid:88) l ti

Original: P ( l ti | h ti ) (cid:2) I 1 ( u ti | l ti ) − I 2 ( u ti | l ti ) (cid:3)(cid:12)(cid:12)(cid:12)

Original: [Equation]

Original: Lemma 2.

Original: [-Let M 
-]
Revised: [-Let M 
-]

Original: i

Original: and M 2 i be two IALMS differing only on their influence distributions

Original: [-I 1 i ( u ti | l ti )and I 
-]
Revised: [-I 1 i ( u ti | l ti )and I 
-]

Original: i ( u ti | l ti ) .

Original: Let Q π i M 1 i and Q π

Original: [-i M 
-]
Revised: [-i M 
-]

Original: i be the value functions induced by M 1 i and M 2 i for the same π i .

Original: If

Original: [-I 1 i and I 
-]
Revised: [-I 1 i and I 
-]

Original: i satisfy

Original: [Equation]

Original: l ti ,

Original: [Equation]

Original: ti

Original: [Equation]

Original: for all π i , h ti , and a ti , where H is the horizon and ¯ R =

Original: ||

Original: R || ∞

Original: Proof.

Original: This is a special case of the simulation lemma (Kearns and Singh, 2002).

Original: We have that the setof local states and actions is the same for both IALMs.

Original: Moreover, the reward function is also thesame R 1 ( x t , a t ) = R 2 ( x t , a t [Equation]
Revised: spent on participant compensation?

Original: [Equation]

Original: ) where P 1 ( h t +1 i | h ti , a ti ) and P 2 ( h t

Original: [-+1 i | h ti , a ti ) are the AOH transitions induced by I 
-]
Revised: [-+1 i | h ti , a ti ) are the AOH transitions induced by I 
-]

Original: and I 2 respectively.

Original: [Equation]

Original: h ti , a ti ) Q π

Original: [-i M 
-]
Revised: [-i M 
-]

Original: i ( h t +1 i , a t +1 i ) − P 2 ( h t

Original: +1 i | h ti , a ti ) Q π

Original: [-i M 
-]
Revised: [-i M 
-]

Original: i ( h t +1 i , a t +1 i ) (cid:105)(cid:12)(cid:12)(cid:12) (cid:12)(cid:12) =

Original: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (cid:88)

Original: h t +1 i ,a t

Original: [-+
-]
Revised: [-+
-]

Original: i π

Original: i ( a t +1 i | h t

Original: +1 i ) (cid:104)

Original: P 1 ( h t +1 i | h ti , a ti ) Q π

Original: [-i M 
-]
Revised: [-i M 
-]

Original: i ( h t +1 i , a t +1 i ) − P 2 ( h t

Original: +1 i | h ti , a ti ) Q π

Original: [-i M 
-]
Revised: [-i M 
-]

Original: i ( h t +1 i , a t +1 i )

Original: + P 2 ( h t

Original: +1 i | h ti , a ti ) Q π

Original: [-i M 
-]
Revised: [-i M 
-]

Original: i ( h t +1 i , a t +1 i ) − P 2 ( h t

Original: +1 i | h ti , a ti ) Q π

Original: [-i M 
-]
Revised: [-i M 
-]

Original: i ( h t +1 i , a t +1 i ) (cid:105)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤ (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)

Original: ¯ R

Original: ( H − t ) (cid:88)

Original: h t +1 i (cid:0) P 1 ( h t

Original: +1 i | h ti , a ti ) − P 2 ( h t +1 i | h ti , a ti ) (cid:1) + (cid:88)

Original: h t +1 i ,a t

Original: [-+
-]
Revised: [-+
-]

Original: i π

Original: i ( a t +1 i | h t

Original: +1 i )

Original: P 2 ( h t +1 i | h ti , a ti )

Original: (cid:104) Q π

Original: [-i M 
-]
Revised: [-i M 
-]

Original: i ( h t +1 i , a

Original: [Equation]

Original: [-17) since Q π i M 
-]
Revised: [-17) since Q π i M 
-]

Original: i

Original: h t +1 i ) ≤ ¯ R ( H − t ) .

Original: Then, from Lemma 3 we know that(cid:88)

Original: [Equation]

Original: [Equation]

Original: Hence,

Original: [Equation]

Original: Theorem 1.

Original: Let M 1 i and M 2 i be two IALMS differing only on their influence distributions

Original: [-I 1 i ( u ti | l ti )and I 
-]
Revised: [-I 1 i ( u ti | l ti )and I 
-]

Original: i ( u ti | l ti ) .

Original: M 1 i and M 2 i induce the same optimal policy π ∗ if, for some ∆ ,

Original: [Equation]

Original: where ¯ a t i = arg max a t i Q

Original: [-π ∗ i M 
-]
Revised: [-π ∗ i M 
-]

Original: i ( h t i , a t i )

Original: Proof.

Original: We will prove it by contradiction.

Original: Let us assume there is a policy π ∗ that is optimal for M 1 ibut not for M 2 i .

Original: This implies that, for some h ti , there is at least one action ˆ a ti ̸

Original: = ¯ a ti for which

Original: [Equation]

Original: [Equation]

Original: Then, because the maximum gap between Q M 1 i and Q M 2 i is ∆ ,

Original: [Equation]

Original: [Equation]

Original: Therefore, we have

Original: [Equation]

Original: which contradicts the statement

Original: [Equation]

Original: B Further Related Work

Original: There is a sizeable body of literature that concentrates on the non-stationarity issues arising fromhaving multiple agents learning simultaneously in the same environment (Laurent et al., 2011;

Original: Hernandez-Leal et al., 2017).

Original: Although often times the problem can be simply ignored with virtuallyno consequences for the agents’ performance (Tan, 1993), in general, disregarding changes in theother agents’ policies, and assuming individual Q-values to be stationary, can have a catastrophiceffect on convergence (Claus and Boutilier, 1998).

Original: The problem of non-stationarity becomes even more severe in the Dec-POMDP setting (Oliehoekand Amato, 2016) since policy changes may not be immediately evident from each agent’s AOH.

Original: Tocompensate for this Raileanu et al.

Original: and Rabinowitz et al.

Original: (2018) explicitly train models thatpredict the other agents’ goals and behaviors.

Original: In contrast, Foerster et al.

Original: (2018a) add an extra term tothe learning objective that is meant to predict the other agents’ parameter updates.

Original: This approach isempirically shown to encourage cooperation in general-sum games.

Original: In order to better approximatethe value function, several works have studied the use of additional information during trainingto inform each individual agent of changes in the other agents’ policies, leading to the ubiquitouscentralized-training decentralized-execution (CTDE) paradigm.

Original: The works by Lowe et al.

Original: (2017)and Foerster et al.

Original: (2018b) exploit this by training a single centralized critic that takes as input thetrue state and joint action of all the agents.

Original: This critic is then used to update the policies of all agentsfollowing the actor-critic policy gradient update (Konda and Tsitsiklis, 1999).

Original: Even though the use ofadditional information to augment the critic may help reduce bias in the value estimates, the idea lacksany theoretical guarantees and has been shown to produce the same policy gradient in expectation asthose produced by multiple independent critics (Lyu et al., 2021).

Original: Moreover, according to Lyu et al.

Original: ,naively augmenting the critic with all other agents’ actions and observations can heavily increasethe variance of the policy gradients.

Original: Both results, however, assume that the critics have converged tothe true on-policy value estimates.

Original: The authors do admit that, in practice, critics are often used evenwhen they have not yet converged.

Original: In such situations, centralized critics might provide more stablepolicy updates since they are better equipped to follow the true non-stationary Q values.

Original: Following asimilar perspective, the concurrent work by Spooner et al.

Original: (2021) tries to reduce variance by using aper-agent baseline function that removes from the policy gradient the contributions to the joint valueestimates of those agents that are conditionally independent, thus effectively providing the agent withmore stable updates.

Original: The works by de Witt et al.

Original: (2020) and Yu et al.

Original: (2021) show that the vanilla

Original: PPO algorithm (Schulman et al., 2017) works already quite well on several multi-agent tasks.

Original: Yu et al.attribute the positive empirical results to the clipping parameter ϵ , which prevents individual policiesfrom changing drastically, and in turn reduces the problem of non-stationarity.

Original: Li et al.

Original: (2021) furtheranalyze this idea and propose a method to estimate the joint policy divergence, which is then used asa constraint in the optimization objective.

Original: C Scope and Limitations

Original: As discussed in the introduction, DIALS targets networked environments with well-defined localregions where the interactions between different regions occur through a limited number of variables.

Original: There is plenty of examples of domains that have this particular structure including traffic networks,heating and water systems, logistics, telecommunications, etc.

Original: Moreover, Knowledge of the influence sources U and how these affect the local regions is required forbuilding the DIALS.

Original: However, we argue that in most cases (as in the two environments we exploredhere) some domain knowledge suffices to be able to tell what the influence sources are.

Original: Having or being able to build high-fidelity local simulators of these local regions is also a requirement.

Original: Fortunately, there exist plenty of simulators of real systems that can readily be used such as, SUMO(Lopez et al., 2018), Robosuite (Zhu et al., 2020), BRAX (Freeman et al., 2021).

Original: There is also a lotof commercial software for building custom-made simulators such as Mujoco (Todorov et al., 2012),or

Original: Unity (Juliani et al., 2018).

Original: Also, note that most (if not all) of the work that has applied RL toreal-world problems relies on simulations to train the policies offline (Bellemare et al., 2020; Degraveet al., 2022).

Original: Hence, we believe our method can have strong impact on many real world applications.

Original: Finally, although the experiments reveal that DIALS can considerably accelerate training times itis also memory demanding.

Original: As shown in Table 3 the total memory usage with DIALS increasesexponentially with the number of simulators/processes.

Original: There is thus a trade-off between fastcomputation and total memory needed.

Original: Note however, that the memory is split among the differentprocesses.

Original: Hence, rather than using a big machine with large memory, DIALS can run on severalsmaller ones with less memory.

Original: D Algorithms

Original: The two algorithms below describe how to generate the datasets { D i } i ∈ N with the GS (Algorithm 2)and how to simulate trajectories with each of the IALS (Algorithm 3).

Original: Algorithm 2 Collect datasets { D i } i ∈ N with GS

Original: Input: T , { ˙ O i } i ∈ N , π 0 = { π 0 i } i ∈ N ▷

Original: Global simulator, observation functions, and joint policy for n ∈ ⟨ 0 , ..., N/T ⟩ do s 0 ← reset ▷ Reset initial state { x 0 i } i ∈ N ← s▷ Extract local states from global state { l 0 i ← x 0 i } i ∈ N ▷ Initialize

Original: [-each agent’s ALSH with initial local state { o 
-]
Revised: [-each agent’s ALSH with initial local state { o 
-]

Original: [-i ∼ O i ( · | x 0 ) } i ∈ N ▷ Sample each agent’s observation from O i { h 
-]
Revised: [-i ∼ O i ( · | x 0 ) } i ∈ N ▷ Sample each agent’s observation from O i { h 
-]

Original: i ← o 0 i } i ∈ N ▷ Initialize each agent’s AOH with initial observation for t ∈ ⟨ 0 , ..., T ⟩ do { u 0 i } i ∈ N ← s▷ Extract each agent’s influence sources from global state { D i ← ( l ti , u ti ) }

Original: i ∈ N ▷ Append ALSH-influence-source pair to the datasets { a ti ∼ π ( · | h ti ) } i ∈ N ▷ Sample each agent’s action from π

Original: i s t +1 ∼ T ( · | s t , a t = { a ti } i ∈ N )

Original: ▷ Sample next state from GS { x t +1 i } i ∈ N ← s t + ▷ Extract local states from global state { l t +1 i ←

Original: ⟨ a ti , x t +1 i ⟩} i ∈ N ▷ Append action-local-state pairs to each agent’s

Original: ALSH { o t +1 i ∼ ˙

Original: O i ( · | x t +1 ) } i ∈ N ▷ Sample each agent’s observation from ˙

Original: O i { h t +1 i ← ⟨ a ti , o t +1 i ⟩}

Original: i ∈ N ▷ Append actions-observation pairs to each agent’s AOH end for end for

Original: Algorithm 3 Simulate agent i ’s trajectory with IALS 1: Input: ˙

Original: T i , ˙

Original: R i , ˙

Original: O i , π i , ˆ I

Original: θ i

Original: ▷ local simulator, local reward and observation functions, policy, AIP 2: x 0 i ← reset ▷ Reset initial state 3: o 0 i ∼ ˙

Original: O i ( ·| x 0 i )

Original: ▷ Sample observation from ˙

Original: [-O i 4: h 0 i ← o 
-]
Revised: [-O i 4: h 0 i ← o 
-]

Original: i ▷ Initialize AOH with initial observation 5: for t ∈ ⟨ 0 , ..., T ⟩ do 6: a ti ∼ π ( · | h ti )

Original: ▷ Sample action 7: ˙

Original: R i ( x ti , a ti )

Original: ▷ Compute reward 8: u ti ∼ ˆ I θ i ( · | l ti )

Original: ▷ Sample influence sources from AIP 9: x t +1 i ∼ ˙ T ( · | x t i , a t i , u t i )

Original: ▷ Sample next local state from LS 10: l t +1 i ← ⟨ a t i , x

Original: t +1 i

Original: ⟩ ▷ Append action-local-state pair to ALSH 11: o t +1 i ∼ ˙

Original: O i ( · | x t +1 i )

Original: ▷ Sample observation from O 12: h t +1 i ← ⟨ a ti , o t

Original: +1 i ⟩ ▷ Append action-observation pair to AOH 13: end for

Original: E

Original: E.1 DIALS vs GS

Original: [Figure]

Original: The plots in Figures 5 and 6 show the learning curves of agents trained with the GS, DIALS, anduntrained-DIALS on the 4 variants of the traffic and warehouse environments (4, 25, 49, and 100agents).

Original: The bar plots show the total runtime of training for 4M timesteps with the three simulators.

Original: Shaded mean.

Original: The orange curves in Figures 5d and 6d stop at 3.5M and 2M timesteps respectively.

Original: This is becausethe maximum execution time allowed by our computer cluster is 1 week, and training 100 agents withthe GS takes longer.

Original: A breakdown of the runtimes for the three simulators is provided in Appendix H.

Original: Note that the runtime measurements were made on the only machine in our computer cluster withmore than 100 CPUs.

Original: This is so that it would fit DIALS when training on the 100-agent variants.

Original: However, the experiments that required less than 100 CPUs were ran on different machines withdifferent CPUs.

Original: The bar plots indicate that DIALS is computationally more efficient and scales much better than the

Original: GS.

Original: Note that the y axis is in log 2 scale.

Original: Moreover, agents trained with DIALS seem to convergesteadily towards similar high-performing policies in both environments, while agents trained withthe GS suffer frequent performance drops and often get stuck in local minima.

Original: This is evidenced bythe oscillations in the orange curves, and the poor mean episodic reward and large standard errorscompared to the green (traffic) and purple (warehouse) curves.

Original: The plots also reveal that estimatingthe influence distributions correctly is important, as indicated by the large gap between DIALS anduntrained-DIALS in both environments.

Original: It is worth noting that the gap between GS and DIALS is larger in the warehouse (Figure 6) thanin the traffic environment (Figure 5).

Original: We posit that this is because, in the warehouse environment,agents are more strongly coupled.

Original: To see this imagine that, by random chance during training, a robotstarts favoring items from one shelf over the three others.

Original: The robot’s neighbors might exploit thisand start collecting items from the unattended shelves.

Original: However, as soon as this first robot changes

Original: [Figure]

Original: A v e r a g e

Original: R e

Original: [Figure]

Original: u

Original: [Figure]

Original: n

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: un

Original: [Figure]

Original: i m

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: A

Original: [Figure]

Original: a g e

Original: R e

Original: [Figure]

Original: u

Original: [Figure]

Original: n

Original: [Figure]

Original: its policy, and starts collecting items more evenly from all four shelves

Original: , the neighbor robots willexperience a sudden drop in the value of their policies, which can have catastrophic effects on thelearning dynamics.

Original: With the DIALS, however, agents are trained on separate simulators and onlybecome aware of changes in the joint policy when the AIPs are retrained.

Original: This prevents them fromconstantly co-adapting to one another.

Original: This is in line with our discussion in Section 4.3.

Original: E.2 AIPs training frequency

Original: The two plots on the left of Figures 7 and 8 show a comparison of the agents’ average return as afunction of runtime for different values of the AIPs training frequency parameter F ( 100 K, 500 K,1 M, and 4 M timesteps).

Original: For ease of visualization, since DIALS F = 500 K, F = 1 M, and F = 4 Mtake shorter to finish than DIALS F = 100 K, the red, green, and purple curves are extended bydotted horizontal lines.

Original: Due to computational limitations, we ran these experiments only on the 4,25, and 49-agent variants of the two environments.

Original: We then chose the best performing values for F( F = 1 M for traffic and F = 4 M for warehouse), and used those to ran DIALS on the environmentswith 100 agents.

Original: In the traffic domain, the gap between the green and the purple curve (Figure 7) suggests that it isimportant to retrain the AIPs at least every 1 M timesteps, such that agents become aware of changesin the other agents’ policies.

Original: This is consistent on all the three variants (Figures 7a, 7b, and 7c).

Original: Incontrast, in the warehouse domain (Figure 8), we see that training the AIPs only once at the beginning(DIALS F = 4 M) is sufficient (Figures 8a, 8b, and 8c).

Original: In fact, as indicated by the gap between thebrown and the rest of the curves, updating the AIPs too frequently (DIALS F = 100 K), aside fromincreasing the runtimes, seems detrimental to the agents’ performance.

Original: This is consistent with ourhypothesis in Section 4.3: “by not updating the AIPs too frequently, we get a biased but otherwisemore consistent learning signal that the agents can rely on to improve their policies.”

Original: The plots on the right of Figures 7 and 8 show the average cross-entropy (CE) loss of the AIPsevaluated on trajectories sampled from the GS.

Original: As explained in Section 4 since all agents learn

Original: [Figure]

Original: simultaneously, the influence distributions { I ( u ti | l ti ) } i ∈ N are non-stationary.

Original: For this reason, we seethat the CE loss changes as the policies of the other agents are updated.

Original: We can also see how the CEloss decreases when the AIPs are retrained, which happens more or less frequently depending onthe hyperparameter F .

Original: Note that the CE not only measures the distance between the two probabilitydistributions but also the absolute entropy.

Original: In the warehouse domain (Figure 8), the neighbor robots’locations become more predictable (lower entropy) as their policies improve.

Original: This explains whythe CE loss decreases even though the AIPs are not updated.

Original: Also note that, in the warehouseenvironment (Figure 8), even though by the end of training DIALS F = 4 M is highly inaccurate, asevidenced by the gap between the purple and the other curves, it is still good enough to train policiesthat match the performance of those trained with DIALS F = 500 K and F = 1 M. This is in linewith our results in Section 4: “Multiple influence distributions may induce the same optimal policy.”

Original: [Figure]

Original: F Implementation Details

Original: F.1 Approximate Influence Predictors

Original: Due to the sequential nature of the problem, rather than feeding the full past history every time wemake a prediction, we use a recurrent neural network (RNN) (Hochreiter and Schmidhuber, 1997;

Original: Cho et al., 2014) and process observations one at a time,

Original: [Equation]

Original: where we use ˆ h to indicate that the history h is embedded in the RNN’s internal memory.

Original: Given that we generally have multiple influence sources u t = ⟨ u 1 t . . .

Original: u Mt ⟩ , we need to fit M separatemodels

Original: ˆ I θ m to predict each of the M influence sources.

Original: In practice, to reduce the computational cost,we can have a single network with a common representation module for all influence sources andoutput their probability distributions using M separate heads.

Original: This representation assumes that theinfluence sources are independent of one another,

Original: [Equation]

Original: which is true for the two domains we study in this paper.

Original: F.2 Local regions

Original: [Figure]
Revised: [N/A]

Original: When choosing the local regions to build the simulators, the only restriction in terms of size is thatthese should contain all the necessary information to compute local observations and rewards.

Original: In ourexperiments, we use one simulator per agent since, given that the simulators run in parallel, this is themost computationally efficient way of factorizing the environment.

Original: Yet, in certain applications, due tohardware limitations (e.g. not enough CPUs or memory available), it might be necessary to partitionthe environment into fewer local regions than the number of agents in the environment.

Original: Moreover, insome environments (including the two we explore here) better results may be obtained by groupingsome of them together in the same simulator.

Original: In fact, one could potentially treat the agents in thesame group/simulator as a single agent and train a policy to control all of them simultaneously.

Original: Note,however, that this is orthogonal to our work as we are mainly concerned with computational speedups.

Original: [Figure]

Original: G Simulators

Original: [Figure]

Original: Figure 9 shows two screenshots of the global simulator (GS) for the traffic (left) and warehouse(right) environments with 25 agents each.

Original: Figure 10 shows two screenshots of the local simulator(LS) for the traffic (left) and warehouse environments (right).

Original: Since all local regions are the same (i.e.˙ T i , ˙

Original: R i , and ˙

Original: O i do not change) in the two environments, we use the same LS for all of them.

Original: However,because depending on where these are located they are influenced differently by the rest of the system,we train separate AIPs, { ˆ I θ i } i ∈ N , for each of them.

Original: Note that, we chose the local regions to be thesame for simplicity.

Original: However, the method can readily be applied to environments with different localtransition dynamics ˙

Original: T i , different local observations ˙

Original: O i , and/or different local rewards ˙

Original: R i for everyagent i ∈ N .

Original: Since all local regions are the same in the two environments, we use the same LS for all of them.

Original: H Runtimes

Original: The two tables below show a breakdown of the runtimes for the two environments and the threesimulators.

Original: These were measured on a machine with 128 CPUs of the type AMD EPYC 7452 32-Core

Original: Processor.

Original: We used this machine for all our measurements because it is the only one in our computercluster that can fit DIALS when training on the 100-agent variants of the environments.

Original: However, theexperiments that required less than 100 CPUs were actually run on different machines.

Original: [Table]

Original: [Table]

Original: I Memory Usage

Original: The table below shows the peak memory usage of the GS and the DIALS.

Original: For the latter we providethe memory usage per process and in total.

Original: The memory needed for the GS increases linearly with thenumber of agents, whereas for DIALS the memory usage per process stays relatively stable.

Original: However,the total amount of memory needed to run DIALS (aggregate of all processes) increases exponentiallywith the number of agents and is considerably larger than that of the GS.

Original: [Table]

Original: J Hyperparameters

Original: The hyperparameters used for the AIPs are reported in Table 4.

Original: Since feeding past local states did notseem to improve the performance of the AIPs in the traffic environment we modeled them with FNNs.

Original: In contrast, adding the past ALSHs does decrease the CE loss in the warehouse environment, andthus we used GRUs (Cho et al., 2014) instead.

Original: The size of the networks was chosen as a compromisebetween low CE loss and computational efficiency.

Original: On the one hand, we need accurate AIPs toproperly capture the influence distributions.

Original: On the other, we also want them to be small enough such that we can make fast predictions.

Original: The hyperparameter named seq.

Original: length determines the number oftimesteps the GRU is backpropagated.

Original: This was chosen to be equal to the horizon such that episodesdid not have to be truncated.

Original: The rest of the hyperparameters in Table 4, which refer to the trainingsetup for the AIPs, were manually tuned.

Original: [Table]

Original: The hyperparmeters used for the policy networks are given in Table 5.

Original: We chose again GRUs forthe warehouse environment and FNNs for the traffic domain, since feeding the previous AOHs didnot seem to improve the agents’ performance in the latter.

Original: The network size and the sequence lengthparameter for the GRUs were manually tuned on the smallest scenarios with 4 agents.

Original: [Table]

Original: [Table]

Original: As for the hyperparameters specific to PPO (Table 6)

Original: [Table]

Original: we used the same values reported by (Schulmanet al., 2017)

Original: [Table]

Original: and only tuned the parameter T , which depends on the rewards and the episode length.

Original: Tdetermines for how many timesteps the value function is rollout before computing the value estimates.

Original: [Table]

Original: [Figure]