Original: M ULTI -A GENT
Revised: M ULTI -A GENT

Original: R
Revised: R

Original: EINFORCEMENT L EARNING
Revised: EINFORCEMENT L EARNING

Original: WITH
Revised: WITH

Original: S HARED R ESOURCE
Revised: S HARED R ESOURCE

Original: FOR I NVENTORY M ANAGEMENT
Revised: FOR I NVENTORY M ANAGEMENT

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We consider inventory management (IM) problem for a single store with a large number of SKUs (stock keeping units) in this paper, where we need to make replenishment decisions for each SKU to balance its supply and demand.
Revised: We consider inventory management (IM) problem for a single store with a large number of SKUs (stock keeping units) in this paper, where we need to make replenishment decisions for each SKU to balance its supply and demand.

Original: Each
Revised: Each

Original: SKU should cooperate with each other to maximize profits, as well as compete for shared resources e.g., warehouse spaces, budget etc.
Revised: SKU should cooperate with each other to maximize profits, as well as compete for shared resources e.g., warehouse spaces, budget etc.

Original: Co-existence of cooperation and competition behaviors makes IM a complicate game, hence IM can be naturally modelled as a multi-agent reinforcement learning (MARL) problem.
Revised: Co-existence of cooperation and competition behaviors makes IM a complicate game, hence IM can be naturally modelled as a multi-agent reinforcement learning (MARL) problem.

Original: In
Revised: In

Original: IM problem, we find that agents only interact indirectly with each other through some shared resources, e.g., warehouse spaces.
Revised: IM problem, we find that agents only interact indirectly with each other through some shared resources, e.g., warehouse spaces.

Original: To formally model MARL problems with above structure, we propose shared resource stochastic game along with an efficient algorithm to learn policies particularly for a large number of agents.
Revised: To formally model MARL problems with above structure, we propose shared resource stochastic game along with an efficient algorithm to learn policies particularly for a large number of agents.

Original: By leveraging shared-resource structure, our method can greatly reduce model complexity and accelerate learning procedure compared with standard MARL algorithms, as shown by extensive experiments.
Revised: By leveraging shared-resource structure, our method can greatly reduce model complexity and accelerate learning procedure compared with standard MARL algorithms, as shown by extensive experiments.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Inventory management (IM) problem has long been one of the most important applications in the supply-chain industry (Nahmias & Smith, 1993).
Revised: Inventory management (IM) problem has long been one of the most important applications in the supply-chain industry (Nahmias & Smith, 1993).

Original: Its main purpose is to maintain a balance between the supply and demand of stock keeping units (SKUs) in a supply chain by optimizing replenishment decisions of each SKU.
Revised: Its main purpose is to maintain a balance between the supply and demand of stock keeping units (SKUs) in a supply chain by optimizing replenishment decisions of each SKU.

Original: Besides leading to profit increment and operational cost reduction, efficient
Revised: Besides leading to profit increment and operational cost reduction, efficient

Original: IM can even give rise to better services to customers.
Revised: IM can even give rise to better services to customers.

Original: However, it is quite a challenging task in practice, especially when there are lots of SKUs involved in the supply-chain.
Revised: However, it is quite a challenging task in practice, especially when there are lots of SKUs involved in the supply-chain.

Original: Particularly, while all
Revised: Particularly, while all

Original: SKUs should cooperate with each other to achieve high profits, they also need to compete for shared resources e.g., warehouse spaces, budget etc.
Revised: SKUs should cooperate with each other to achieve high profits, they also need to compete for shared resources e.g., warehouse spaces, budget etc.

Original: Such co-existence of cooperation and competition renders IM a complicated game that is hard to address.
Revised: Such co-existence of cooperation and competition renders IM a complicated game that is hard to address.

Original: Traditional methods usually reduce IM problems to solving dynamic programming problems.
Revised: Traditional methods usually reduce IM problems to solving dynamic programming problems.

Original: However, these approaches often rely on some unrealistic assumptions such as i.i.d.
Revised: However, these approaches often rely on some unrealistic assumptions such as i.i.d.

Original: demand, deterministic leading time, etc.
Revised: demand, deterministic leading time, etc.

Original: Moreover, as the state space grows exponentially along with some key factors like leading time and number of SKUs (Gijsbrechts et al., 2019), the corresponding dynamic programming problems become intractable due to the curse of dimensionality.
Revised: Moreover, as the state space grows exponentially along with some key factors like leading time and number of SKUs (Gijsbrechts et al., 2019), the corresponding dynamic programming problems become intractable due to the curse of dimensionality.

Original: Because of these limitations, many approaches based on approximate dynamic programming are proposed to solve IM problems in different settings (Halman et al., 2009; Fang et al., 2013; Chen & Yang, 2019).
Revised: Because of these limitations, many approaches based on approximate dynamic programming are proposed to solve IM problems in different settings (Halman et al., 2009; Fang et al., 2013; Chen & Yang, 2019).

Original: While these approaches perform well in certain scenarios, they heavily rely on problem-specific expertise or assumptions e.g., the zero or one period leading time assumption in (Halman et al., 2009), hence can hardly generalize to other settings.
Revised: While these approaches perform well in certain scenarios, they heavily rely on problem-specific expertise or assumptions e.g., the zero or one period leading time assumption in (Halman et al., 2009), hence can hardly generalize to other settings.

Original: In contrast, reinforcement learning (RL) based methods, with fast inference speed, can generalize to various scenarios in a data-driven manner.
Revised: In contrast, reinforcement learning (RL) based methods, with fast inference speed, can generalize to various scenarios in a data-driven manner.

Original: However, it is usually too costly to train a global policy that can make decisions for all SKUs, since the training efficiency can be notably curtailed because of the large global state and action space (Jiang & Agarwal, 2018)).
Revised: However, it is usually too costly to train a global policy that can make decisions for all SKUs, since the training efficiency can be notably curtailed because of the large global state and action space (Jiang & Agarwal, 2018)).

Original: To further address the training efficiency issue, it is natural to adopt the multi-agent reinforcement learning (MARL) paradigm, where each SKU is regarded as an agent whose state and action spaces are localized and only contain information relevant to itself.
Revised: To further address the training efficiency issue, it is natural to adopt the multi-agent reinforcement learning (MARL) paradigm, where each SKU is regarded as an agent whose state and action spaces are localized and only contain information relevant to itself.

Original: There are currently two popular paradigms to train MARL in the literature: independent learning (Tan, 1993) and joint action learning (Lowe et al., 2017).
Revised: There are currently two popular paradigms to train MARL in the literature: independent learning (Tan, 1993) and joint action learning (Lowe et al., 2017).

Original: Despite of their success in many scenarios, these two MARL paradigms also exhibit certain weaknesses that restrain their effectiveness in solving IM problems.
Revised: Despite of their success in many scenarios, these two MARL paradigms also exhibit certain weaknesses that restrain their effectiveness in solving IM problems.

Original: On one hand, if applying independent learning, policy training of one agent simply treats all other agents as parts of the stochastic environment, hence is hard to converge due to non-stationary of the environment.
Revised: On one hand, if applying independent learning, policy training of one agent simply treats all other agents as parts of the stochastic environment, hence is hard to converge due to non-stationary of the environment.

Original: On the other hand, joint action learning usually learns a centralized critic conditioned on the joint action and state spaces of all agents, which can easily become intractable with increasing number of SKUs.
Revised: On the other hand, joint action learning usually learns a centralized critic conditioned on the joint action and state spaces of all agents, which can easily become intractable with increasing number of SKUs.

Original: Furthermore, it could be quite time-consuming to sample data from joint simulator for a great number of SKUs, since it usually involves much computation on many internal variables caused by complex agent interactions.
Revised: Furthermore, it could be quite time-consuming to sample data from joint simulator for a great number of SKUs, since it usually involves much computation on many internal variables caused by complex agent interactions.

Original: To address these challenges, we took a closer look at the IM problem and find that there exists a special structure that can be leveraged to design more effective MARL paradigm.
Revised: To address these challenges, we took a closer look at the IM problem and find that there exists a special structure that can be leveraged to design more effective MARL paradigm.

Original: Particularly, each agent in IM only interacts with others through the shared resource, e.g., warehouse spaces.
Revised: Particularly, each agent in IM only interacts with others through the shared resource, e.g., warehouse spaces.

Original: We introduce an auxiliary variable to represent the whole inventory level, implying the available shared resource, for all SKUs, and refer to this variable as context .
Revised: We introduce an auxiliary variable to represent the whole inventory level, implying the available shared resource, for all SKUs, and refer to this variable as context .

Original: From the MARL perspective, one agent can be influenced by other agents only through such context.
Revised: From the MARL perspective, one agent can be influenced by other agents only through such context.

Original: The context dynamics actually reflect the collective behaviors of all agents.
Revised: The context dynamics actually reflect the collective behaviors of all agents.

Original: And, conditioned on context dynamics, one agent’s state transition and reward function are independent of other agents.
Revised: And, conditioned on context dynamics, one agent’s state transition and reward function are independent of other agents.

Original: In this way, leveraging context as an additional input for each agent’s policy/value functions enables us to both avoid the non-stationary problem caused by independent learning and mitigate the intractable centralized critics learning caused by exploding state and action spaces.
Revised: In this way, leveraging context as an additional input for each agent’s policy/value functions enables us to both avoid the non-stationary problem caused by independent learning and mitigate the intractable centralized critics learning caused by exploding state and action spaces.

Original: Moreover, introducing context dynamics inspires us to build a local simulator for each agent in order to facilitate more efficient policy learning for each agent.
Revised: Moreover, introducing context dynamics inspires us to build a local simulator for each agent in order to facilitate more efficient policy learning for each agent.

Original: Based on this structure with context dynamics, we propose a shared-resource stochastic game to model the IM problem.
Revised: Based on this structure with context dynamics, we propose a shared-resource stochastic game to model the IM problem.

Original: Specifically, we introduce a surrogate objective function to optimize the return of agent i conditioned on all other agents, denoted by − i .
Revised: Specifically, we introduce a surrogate objective function to optimize the return of agent i conditioned on all other agents, denoted by − i .

Original: Here, we make two approximations to get the surrogate objective function: 1) rearranging the sampling process by first sampling context dynamics then sampling local state/action/reward for each agent i ; 2) using context dynamics sampled by previous policies.
Revised: Here, we make two approximations to get the surrogate objective function: 1) rearranging the sampling process by first sampling context dynamics then sampling local state/action/reward for each agent i ; 2) using context dynamics sampled by previous policies.

Original: Based on above surrogate objective, our algorithm consists of two iterative learning procedures: 1) obtaining context dynamics from joint simulator, 2) updating policy for each agent by data sampled from its respective local simulator conditioned on collective context dynamics.
Revised: Based on above surrogate objective, our algorithm consists of two iterative learning procedures: 1) obtaining context dynamics from joint simulator, 2) updating policy for each agent by data sampled from its respective local simulator conditioned on collective context dynamics.

Original: By decoupling each agent from all others with a separate training procedure, our method can greatly reduce model complexity and accelerate learning procedure, as shown by extensive experiments.
Revised: By decoupling each agent from all others with a separate training procedure, our method can greatly reduce model complexity and accelerate learning procedure, as shown by extensive experiments.

Original: It is worthwhile to mention that our method is not limited to IM, but can also be applied to many other applications with shared-resource structure, e.g., portfolio management (Ye et al., 2020), smart grid scheduling (Remani et al., 2019), etc.
Revised: It is worthwhile to mention that our method is not limited to IM, but can also be applied to many other applications with shared-resource structure, e.g., portfolio management (Ye et al., 2020), smart grid scheduling (Remani et al., 2019), etc.

Original: Our contributions are summarized as follows:
Revised: Our contributions are summarized as follows:

Original: • We propose shared-resource stochastic game to capture the problem structure in IM, that agents only interact with each other through shared resource.
Revised: • We propose shared-resource stochastic game to capture the problem structure in IM, that agents only interact with each other through shared resource.

Original: • We propose a novel algorithm that leverages the shared-resource structure to solve IM problem efficiently.
Revised: • We propose a novel algorithm that leverages the shared-resource structure to solve IM problem efficiently.

Original: • Extensive experiments demonstrate that our method outperforms existing MARL algorithms on both profits and computing efficiency.
Revised: • Extensive experiments demonstrate that our method outperforms existing MARL algorithms on both profits and computing efficiency.

Original:  B ACKGROUND 2. S TOCHASTIC G AMES
Revised:  B ACKGROUND 2. S TOCHASTIC G AMES

Original: We build our work on stochastic games (SGs) (Shapley, 1953), since each SKU in IM problem has its own profit (reward) to optimize.
Revised: We build our work on stochastic games (SGs) (Shapley, 1953), since each SKU in IM problem has its own profit (reward) to optimize.

Original: A Stochastic Game is defined by a tuple (cid:16)
Revised: A Stochastic Game is defined by a tuple (cid:16)

Original: N , S , (cid:8)
Revised: N , S , (cid:8)

Original: A i (cid:9) i ∈N , T , (cid:8) R i (cid:9) i ∈N , γ (cid:17) , where N = { 1 , · · · , n } denotes the set of n > 1 agents,
Revised: A i (cid:9) i ∈N , T , (cid:8) R i (cid:9) i ∈N , γ (cid:17) , where N = { 1 , · · · , n } denotes the set of n > 1 agents,

Original: S denotes the state space observed by all agents, A i denotes the action space of agent i .
Revised: S denotes the state space observed by all agents, A i denotes the action space of agent i .

Original: Let
Revised: Let

Original: A :
Revised: A :

Original: = A 1 × · · · × A n , then T : S ×
Revised: = A 1 × · · · × A n , then T : S ×

Original: A → ∆( S ) denotes the transition probability from any state s ∈ S to any state s ′ ∈ S after taking a joint action a ∈ A ; R i : S ×
Revised: A → ∆( S ) denotes the transition probability from any state s ∈ S to any state s ′ ∈ S after taking a joint action a ∈ A ; R i : S ×

Original: A × S → R is the reward function that determines the immediate reward received by agent i for a transition from ( s, a ) to s ′ ; γ ∈ [0 , 1) is the discount factor.
Revised: A × S → R is the reward function that determines the immediate reward received by agent i for a transition from ( s, a ) to s ′ ; γ ∈ [0 , 1) is the discount factor.

Original: We can formulate the joint policy of other agents’ as π − i =
Revised: We can formulate the joint policy of other agents’ as π − i =

Original: (cid:81)
Revised: (cid:81)

Original: j ∈− i π j .
Revised: j ∈− i π j .

Original: Each agent i optimizes its policy
Revised: Each agent i optimizes its policy

Original: π i : S → ∆ (cid:0) A i (cid:1) to maximize its own long-term reward, which is conditioned on other agents’ behavior, defined as
Revised: π i : S → ∆ (cid:0) A i (cid:1) to maximize its own long-term reward, which is conditioned on other agents’ behavior, defined as

Original: [Equation]
Revised: [Equation]

Original: We will illustrate the shared resource structure of IM problem in Section 2.2, which motivates us to propose shared-resource stochastic game as a special case of stochastic game to capture such structure in Section 3.1.
Revised: We will illustrate the shared resource structure of IM problem in Section 2.2, which motivates us to propose shared-resource stochastic game as a special case of stochastic game to capture such structure in Section 3.1.

Original: 2. I NVENTORY M ANAGEMENT WITH S HARED R ESOURCE
Revised: 2. I NVENTORY M ANAGEMENT WITH S HARED R ESOURCE

Original: While a typical setting for IM shall involve a supply network of multi-echelon including stores, warehouses and factories, we simplify the setting to ease our presentation.
Revised: While a typical setting for IM shall involve a supply network of multi-echelon including stores, warehouses and factories, we simplify the setting to ease our presentation.

Original: In the following, we shall focus on scenarios with one store and multiple SKUs.
Revised: In the following, we shall focus on scenarios with one store and multiple SKUs.

Original: We further assume that there is an upstream warehouse that can fulfill requirements from the store perfectly.
Revised: We further assume that there is an upstream warehouse that can fulfill requirements from the store perfectly.

Original: Our objective is to learn high-quality replenishing policies for each SKU in the store, particularly when there are a large number of SKUs.
Revised: Our objective is to learn high-quality replenishing policies for each SKU in the store, particularly when there are a large number of SKUs.

Original: As replenishing decisions for SKUs in stores should directly consider consumption behaviors of customers and competitions from other SKUs due to limited resources like spaces, budget etc.
Revised: As replenishing decisions for SKUs in stores should directly consider consumption behaviors of customers and competitions from other SKUs due to limited resources like spaces, budget etc.

Original: , they are more challenging to optimize comparing to SKUs in warehouses.
Revised: , they are more challenging to optimize comparing to SKUs in warehouses.

Original: It is worthwhile to mention that, due to the flexibility of RL algorithms, our method can also be applied to more complex settings with multi-echelon, fluctuate supply, non-deterministic leading time etc.
Revised: It is worthwhile to mention that, due to the flexibility of RL algorithms, our method can also be applied to more complex settings with multi-echelon, fluctuate supply, non-deterministic leading time etc.

Original: Similar to previous work, we follow the multi-agent RL (MARL) framework with decentralized agents, each of which manages inventory of one SKU in the store.
Revised: Similar to previous work, we follow the multi-agent RL (MARL) framework with decentralized agents, each of which manages inventory of one SKU in the store.

Original: We assume that the store has n
Revised: We assume that the store has n

Original: SKUs in sell, all of which share a common space that can store up to I max units at the same time.
Revised: SKUs in sell, all of which share a common space that can store up to I max units at the same time.

Original: Replenishing decisions of each SKU are made on discretized time steps, which are days in our paper.
Revised: Replenishing decisions of each SKU are made on discretized time steps, which are days in our paper.

Original: For each time step t and SKU i , let ˙
Revised: For each time step t and SKU i , let ˙

Original: I it ∈
Revised: I it ∈

Original: Z denote units of i that are in stock.
Revised: Z denote units of i that are in stock.

Original: Hence, the following constraint shall hold for all time steps:
Revised: Hence, the following constraint shall hold for all time steps:

Original: [Equation]
Revised: [Equation]

Original: At any time step t , the agent corresponding to SKU i may place a replenishment order to request
Revised: At any time step t , the agent corresponding to SKU i may place a replenishment order to request

Original: O it ∈
Revised: O it ∈

Original: Z units of products from its upstream warehouse.
Revised: Z units of products from its upstream warehouse.

Original: These replenishment orders cannot be fulfillment instantly, but will take several time steps, referred as leading time, before these products are delivered to the store.
Revised: These replenishment orders cannot be fulfillment instantly, but will take several time steps, referred as leading time, before these products are delivered to the store.

Original: Let L i denote the leading time of SKU i and T it
Revised: Let L i denote the leading time of SKU i and T it

Original: ∈
Revised: ∈

Original: Z its total units in transit at the time step t .
Revised: Z its total units in transit at the time step t .

Original: Meanwhile, demands from customers D it may consume inventory of SKU i and cause an actual sale of units S it ∈
Revised: Meanwhile, demands from customers D it may consume inventory of SKU i and cause an actual sale of units S it ∈

Original: Z .
Revised: Z .

Original: Due to the possibility of out-of-stock, S it may be less than D it .
Revised: Due to the possibility of out-of-stock, S it may be less than D it .

Original: Formally, dynamics of these variables can be described as follows:
Revised: Formally, dynamics of these variables can be described as follows:

Original: [Equation]
Revised: [Equation]

Original: As we mentioned before, due to all SKUs share a common space, it may happen that the storage overflows when ordered products arrive.
Revised: As we mentioned before, due to all SKUs share a common space, it may happen that the storage overflows when ordered products arrive.

Original: In this paper, we assume that the excess SKUs will be discarded proportionally according to ρ defined in Eq. 6, which corresponds to the overflowing ratio if we accept all coming SKUs without considering the space constraint.
Revised: In this paper, we assume that the excess SKUs will be discarded proportionally according to ρ defined in Eq. 6, which corresponds to the overflowing ratio if we accept all coming SKUs without considering the space constraint.

Original: To calculate ρ , we also introduce an extra variable
Revised: To calculate ρ , we also introduce an extra variable

Original: ˆ
Revised: ˆ

Original: I it , which we call afterstate of ˙
Revised: I it , which we call afterstate of ˙

Original: I it .
Revised: I it .

Original: Intuitively, it denotes units of SKU i in stock at the end of the t -th time step if we omit the capacity constraint.
Revised: Intuitively, it denotes units of SKU i in stock at the end of the t -th time step if we omit the capacity constraint.

Original: We shall note that other manners e.g., prioritizing all SKUs, to resolve space overflows are possible.
Revised: We shall note that other manners e.g., prioritizing all SKUs, to resolve space overflows are possible.

Original: The algorithm that we will introduce in the following section will also apply to these settings.
Revised: The algorithm that we will introduce in the following section will also apply to these settings.

Original: Undoubtedly, such behaviors will cause extra operational cost which should be avoided as much as possible in our replenishing decisions.
Revised: Undoubtedly, such behaviors will cause extra operational cost which should be avoided as much as possible in our replenishing decisions.

Original: The intermediate profit Pt it of the i -th SKU is calculated according to the following equation:
Revised: The intermediate profit Pt it of the i -th SKU is calculated according to the following equation:

Original: [Equation]
Revised: [Equation]

Original: where p i and q i are the unit sale price, unit procurement price for the i -th SKU, respectively, and
Revised: where p i and q i are the unit sale price, unit procurement price for the i -th SKU, respectively, and

Original: o and h are the order cost and unit holding cost, respectively.
Revised: o and h are the order cost and unit holding cost, respectively.

Original: I [ · ] is an indicator function which equals to one when the condition is true and zero otherwise.
Revised: I [ · ] is an indicator function which equals to one when the condition is true and zero otherwise.

Original: The order cost reflects the fixed transportation cost or the order processing cost, and yields whenever the order quantity is non-zero.
Revised: The order cost reflects the fixed transportation cost or the order processing cost, and yields whenever the order quantity is non-zero.

Original: For convenience, we summarize all notations in Table 2 in Appendix B, where we also give an example with 2 SKUs in Fig.
Revised: For convenience, we summarize all notations in Table 2 in Appendix B, where we also give an example with 2 SKUs in Fig.

Original: 4 to further illustrate the whole procedure.
Revised: 4 to further illustrate the whole procedure.

Original:  M ETHODOLOGY 3. S HARED - RESOURCE S TOCHASTIC G AME
Revised:  M ETHODOLOGY 3. S HARED - RESOURCE S TOCHASTIC G AME

Original: In this section, we show that IM problem introduced in Section 2.2 can be formulated as a shared-resource stochastic game, where each agent is only influenced by other agents through a shared resource pool.
Revised: In this section, we show that IM problem introduced in Section 2.2 can be formulated as a shared-resource stochastic game, where each agent is only influenced by other agents through a shared resource pool.

Original: We define a shared-resource stochastic game as a tuple (cid:16)
Revised: We define a shared-resource stochastic game as a tuple (cid:16)

Original: N , (cid:8) S
Revised: N , (cid:8) S

Original: i (cid:9) i ∈N , C , (cid:8) A i (cid:9)
Revised: i (cid:9) i ∈N , C , (cid:8) A i (cid:9)

Original: i ∈N , T , (cid:8) R i (cid:9) i ∈N , γ (cid:17) , where N = { 1 , · · · , n } denotes the set of n >agents, S
Revised: i ∈N , T , (cid:8) R i (cid:9) i ∈N , γ (cid:17) , where N = { 1 , · · · , n } denotes the set of n >agents, S

Original: i denotes the state space of agent i , C denotes the context space observed by all agents, A i denotes the action space of agent i .
Revised: i denotes the state space of agent i , C denotes the context space observed by all agents, A i denotes the action space of agent i .

Original: Here, the context represents the occupied capacity of sharedresource.
Revised: Here, the context represents the occupied capacity of sharedresource.

Original: Let S :
Revised: Let S :

Original: = S 1 × · · · × S n and A :
Revised: = S 1 × · · · × S n and A :

Original: =
Revised: =

Original: A 1 × · · · × A n , then T : S × C ×
Revised: A 1 × · · · × A n , then T : S × C ×

Original: A → ∆( S × C ) denotes the transition probability, which can be decomposed as follows.
Revised: A → ∆( S × C ) denotes the transition probability, which can be decomposed as follows.

Original: The context is affected by all agents, i.e., c t +1 ∼ P c ( · | c t , s t , a t ) .
Revised: The context is affected by all agents, i.e., c t +1 ∼ P c ( · | c t , s t , a t ) .

Original: Since we are dealing with resource, the transition function of context usually has some additive structure with respect to all agents, which we will illustrate later in Section 3.2.
Revised: Since we are dealing with resource, the transition function of context usually has some additive structure with respect to all agents, which we will illustrate later in Section 3.2.

Original: Given context c t +1 , the transition function of state for each agent is independent with other agents, i.e., s it +1 ∼ P is ( · | s it , a it , c t +1 ) .
Revised: Given context c t +1 , the transition function of state for each agent is independent with other agents, i.e., s it +1 ∼ P is ( · | s it , a it , c t +1 ) .

Original: Reward function is also independent with other agents given context, r it ∼ R
Revised: Reward function is also independent with other agents given context, r it ∼ R

Original: i ( s it , a it , c t +1 ) .
Revised: i ( s it , a it , c t +1 ) .

Original: We refer P is and R i as local transition and reward functions.
Revised: We refer P is and R i as local transition and reward functions.

Original: γ ∈ [0 , 1) is the discount factor.
Revised: γ ∈ [0 , 1) is the discount factor.

Original: Each agent i optimizes its own policy
Revised: Each agent i optimizes its own policy

Original: π i : S i × C → ∆ (cid:0)
Revised: π i : S i × C → ∆ (cid:0)

Original: A i (cid:1) to maximize the expected long-term reward conditioned on other agents’ policies π − i , defined as
Revised: A i (cid:1) to maximize the expected long-term reward conditioned on other agents’ policies π − i , defined as

Original: [Equation]
Revised: [Equation]

Original: Given the above definition, an IM problem can be formulated as a shared-resource stochastic game by letting r it =
Revised: Given the above definition, an IM problem can be formulated as a shared-resource stochastic game by letting r it =

Original: Pt it , a it
Revised: Pt it , a it

Original: = O it , and c t = (cid:80) ni
Revised: = O it , and c t = (cid:80) ni

Original: =1 ˙
Revised: =1 ˙

Original: I it for all SKU i and time step t .
Revised: I it for all SKU i and time step t .

Original: Moreover, state of each SKU i will be a concatenation of information like T it , p i , its historical actions and demands etc.
Revised: Moreover, state of each SKU i will be a concatenation of information like T it , p i , its historical actions and demands etc.

Original: A detailed description of the state space can be found in the Appendix E.3.
Revised: A detailed description of the state space can be found in the Appendix E.3.

Original: 3. S URROGATE O BJECTIVE WITH C ONTEXT D YNAMICS
Revised: 3. S URROGATE O BJECTIVE WITH C ONTEXT D YNAMICS

Original: We now introduce how to optimize the return for agent i conditioned on other agents’ behaviors.
Revised: We now introduce how to optimize the return for agent i conditioned on other agents’ behaviors.

Original: Roughly speaking, the context dynamics reflect the collective behaviors of other agents.
Revised: Roughly speaking, the context dynamics reflect the collective behaviors of other agents.

Original: It is possible to estimate the objective function approximately for each agent i only based on its local transition dynamics, local reward function and the context dynamics.
Revised: It is possible to estimate the objective function approximately for each agent i only based on its local transition dynamics, local reward function and the context dynamics.

Original: First, we give a detailed description of the transition model of shared resource c from each agent’s perspective.
Revised: First, we give a detailed description of the transition model of shared resource c from each agent’s perspective.

Original: Given such dynamics for the shared resource, we then show how to approximate the objective function in Eq. 9 by rearranging the sampling process.
Revised: Given such dynamics for the shared resource, we then show how to approximate the objective function in Eq. 9 by rearranging the sampling process.

Original: Finally, we replace the policies for sampling context dynamics by old policies from the previous iteration to accelerate the decentralized training.
Revised: Finally, we replace the policies for sampling context dynamics by old policies from the previous iteration to accelerate the decentralized training.

Original: 3.2. C ONTEXT D YNAMICS AND L OCAL S IMULATOR
Revised: 3.2. C ONTEXT D YNAMICS AND L OCAL S IMULATOR

Original: We use ˙
Revised: We use ˙

Original: c it to represent the amount of resource occupied by agent i at time step t and ˙ c t the total amount of occupied resource.
Revised: c it to represent the amount of resource occupied by agent i at time step t and ˙ c t the total amount of occupied resource.

Original: We further let ˙ c − i t denote the amount of resource occupied by all agents but i .
Revised: We further let ˙ c − i t denote the amount of resource occupied by all agents but i .

Original: Since the capacity has the additive structure, we have the following equations:
Revised: Since the capacity has the additive structure, we have the following equations:

Original: [Equation]
Revised: [Equation]

Original: Similarly, we denote the afterstate of ˙
Revised: Similarly, we denote the afterstate of ˙

Original: c it as ˆ c it .
Revised: c it as ˆ c it .

Original: From the perspective of agent i , it can view the context ˙
Revised: From the perspective of agent i , it can view the context ˙

Original: c − i t as a part of the environment.
Revised: c − i t as a part of the environment.

Original: Hence, given s it , ˙ c it , and a it , P (ˆ c it | s it , a it , ˙
Revised: Hence, given s it , ˙ c it , and a it , P (ˆ c it | s it , a it , ˙

Original: c it ) represents how the replenishment decision influences ˆ c it .
Revised: c it ) represents how the replenishment decision influences ˆ c it .

Original: Due to the additive structure, we have ˆ c t
Revised: Due to the additive structure, we have ˆ c t

Original: = ˆ c
Revised: = ˆ c

Original: it + ˆ c − i t .
Revised: it + ˆ c − i t .

Original: Then by applying a resource overflow resolution procedure as in Eq.
Revised: Then by applying a resource overflow resolution procedure as in Eq.

Original: 6, we obtain ˙
Revised: 6, we obtain ˙

Original: c it +1 representing state of the resource in the next step.
Revised: c it +1 representing state of the resource in the next step.

Original: We use notation c t to represent (ˆ c t − 1 , ˙ c t ) .
Revised: We use notation c t to represent (ˆ c t − 1 , ˙ c t ) .

Original: We refer ( s i , c i ) as the state for agent i, and c − i as the context.
Revised: We refer ( s i , c i ) as the state for agent i, and c − i as the context.

Original: For each agent, we have the following sampling process T i given the context dynamics of c − i ,
Revised: For each agent, we have the following sampling process T i given the context dynamics of c − i ,

Original: [Equation]
Revised: [Equation]

Original: Given the context dynamics for c − i , we can build a local simulator for agent i according to the above equations.
Revised: Given the context dynamics for c − i , we can build a local simulator for agent i according to the above equations.

Original: To ease our presentation, we only consider one kind of shared resource i.e., warehouse spaces in the above formulation.
Revised: To ease our presentation, we only consider one kind of shared resource i.e., warehouse spaces in the above formulation.

Original: However, it can be extended to support multiple resources as long as these resources have the additive structure as in Eq.
Revised: However, it can be extended to support multiple resources as long as these resources have the additive structure as in Eq.

Original: 3.2. S URROGATE L OSS F UNCTION
Revised: 3.2. S URROGATE L OSS F UNCTION

Original: To leverage local simulators, we rearrange the sampling process for evaluating η ( π i , π − i ) as follows.
Revised: To leverage local simulators, we rearrange the sampling process for evaluating η ( π i , π − i ) as follows.

Original: First, we follow the regular sampling process in Eq. 9, and get samples of c − i t .
Revised: First, we follow the regular sampling process in Eq. 9, and get samples of c − i t .

Original: Then, we re-sample
Revised: Then, we re-sample

Original: ( s it , a it , c it ) based on samples of c − i t following Eq.
Revised: ( s it , a it , c it ) based on samples of c − i t following Eq.

Original: It is worth noting that we make an approximation here.
Revised: It is worth noting that we make an approximation here.

Original: Given samples for c − i t , we actually need to inference the posterior of ( s it , a it , c it ) .
Revised: Given samples for c − i t , we actually need to inference the posterior of ( s it , a it , c it ) .

Original: However, since we consider scenarios with lots of agents, it is reasonable to assume that each agent i has limited influence on its context c − i .
Revised: However, since we consider scenarios with lots of agents, it is reasonable to assume that each agent i has limited influence on its context c − i .

Original: Therefore, we can assume that c − i t has limited influence on the inference of ( s it , a it , c it ) and sample the latter directly according to Eq.
Revised: Therefore, we can assume that c − i t has limited influence on the inference of ( s it , a it , c it ) and sample the latter directly according to Eq.

Original: By rearranging the sampling process, we obtain the following surrogate objective,
Revised: By rearranging the sampling process, we obtain the following surrogate objective,

Original: [Equation]
Revised: [Equation]

Original: In practice, it is quite costly to sample from the joint dynamics T , but much cheaper to sample from the local dynamics T i .
Revised: In practice, it is quite costly to sample from the joint dynamics T , but much cheaper to sample from the local dynamics T i .

Original: To leverage data samples efficiently, we propose to use the samples from previous policies in our objective function, which is a common trick in reinforcement learning.
Revised: To leverage data samples efficiently, we propose to use the samples from previous policies in our objective function, which is a common trick in reinforcement learning.

Original: Specifically, we use samples of c − i t collected by policies ( π iold , π − i old ) of the previous iteration, and further rewrite the surrogate objective function as follows:
Revised: Specifically, we use samples of c − i t collected by policies ( π iold , π − i old ) of the previous iteration, and further rewrite the surrogate objective function as follows:

Original: [Equation]
Revised: [Equation]

Original: As long as current policies stay close to previous ones, we can adopt above surrogate objective function to improve sample efficiency.
Revised: As long as current policies stay close to previous ones, we can adopt above surrogate objective function to improve sample efficiency.

Original: 3. A LGORITHM
Revised: 3. A LGORITHM

Original: In the following, we will present details about our proposed algorithm, which is referred as Contextaware Decentralized PPO (CD-PPO).
Revised: In the following, we will present details about our proposed algorithm, which is referred as Contextaware Decentralized PPO (CD-PPO).

Original: We call it context-aware as the context c − i t is a part of input to train policy and value functions of each agent.
Revised: We call it context-aware as the context c − i t is a part of input to train policy and value functions of each agent.

Original: Such a context-aware approach can avoid the nonstationary problem occurred in independent learning methods (which does not consider dynamics of others during training), since the context reflects other agents’ collective behaviors which can impact dynamics of each individual agent.
Revised: Such a context-aware approach can avoid the nonstationary problem occurred in independent learning methods (which does not consider dynamics of others during training), since the context reflects other agents’ collective behaviors which can impact dynamics of each individual agent.

Original: In the meanwhile, our method can mitigate the intractable centralized critic learning by avoiding using exploding joint state and action space, i.e., ( s t , a t , c t ) , as input of the critic.
Revised: In the meanwhile, our method can mitigate the intractable centralized critic learning by avoiding using exploding joint state and action space, i.e., ( s t , a t , c t ) , as input of the critic.

Original: As shown in Algorithm 1, our algorithm consists of two iterative learning procedures: 1) obtaining context dynamics by running all agents in the joint environment (Algorithm 2); 2) updating policy of each agent using data sampled from its local simulator conditioned on the context dynamics (Algorithm 3).
Revised: As shown in Algorithm 1, our algorithm consists of two iterative learning procedures: 1) obtaining context dynamics by running all agents in the joint environment (Algorithm 2); 2) updating policy of each agent using data sampled from its local simulator conditioned on the context dynamics (Algorithm 3).

Original: Moreover, our algorithm follows a decentralized training paradigm with an acceptable communication cost of the context dynamics.
Revised: Moreover, our algorithm follows a decentralized training paradigm with an acceptable communication cost of the context dynamics.

Original: We refer readers to Appendix C for a detailed description.
Revised: We refer readers to Appendix C for a detailed description.

Original: It is worth noting that a naive approach is to optimize policies by Eq.
Revised: It is worth noting that a naive approach is to optimize policies by Eq.

Original: 9 with data sampled from the joint simulator.
Revised: 9 with data sampled from the joint simulator.

Original: Nonetheless, we find that it is empirically more time-consuming to sample one step in the joint simulator than letting each of the n local simulators sample once.
Revised: Nonetheless, we find that it is empirically more time-consuming to sample one step in the joint simulator than letting each of the n local simulators sample once.

Original: One major reason lies in that agent interactions take most of the simulation time.
Revised: One major reason lies in that agent interactions take most of the simulation time.

Original: Our method, with the advantage of leveraging local simulators to simplify interactions, are henceforth much cheaper to sample.
Revised: Our method, with the advantage of leveraging local simulators to simplify interactions, are henceforth much cheaper to sample.

Original: We refer readers to Appendix D for more discussions on the benefit of leveraging local simulators.
Revised: We refer readers to Appendix D for more discussions on the benefit of leveraging local simulators.

Original: Algorithm

Original: Given the joint simulator Env joint and local simulators { Env ilocal } ni =
Revised: Given the joint simulator Env joint and local simulators { Env ilocal } ni =

Original: Initialize policies π i and value functions V i for i = 1 , . . .
Revised: Initialize policies π i and value functions V i for i = 1 , . . .

Original: , n for M epochs do // Collect context dynamics via running joint simulation { c − 1 t } Tt =0 , . .
Revised: , n for M epochs do // Collect context dynamics via running joint simulation { c − 1 t } Tt =0 , . .

Original: , { c − n t } Tt =

Revised: , { c − n t } Tt =

Original: ← GetContextDynamics ( Env joint , { π i } ni =1 ) (Algorithm 2) for k = 1 , 2 . . .
Revised: ← GetContextDynamics ( Env joint , { π i } ni =1 ) (Algorithm 2) for k = 1 , 2 . . .

Original: , K do for all agents i do //
Revised: , K do for all agents i do //

Original: Set capacity trajectory by context dynamics
Revised: Set capacity trajectory by context dynamics

Original: Env ilocal .
Revised: Env ilocal .

Original: set c trajectory ( { c − i t } Tt =0 ) //
Revised: set c trajectory ( { c − i t } Tt =0 ) //

Original: Train policy by running simulation in the corresponding local environment π i , V
Revised: Train policy by running simulation in the corresponding local environment π i , V

Original: i ← DecentralizedPPO
Revised: i ← DecentralizedPPO

Original: ( Env ilocal , π i , V i ) (Algorithm 3) end for end for
Revised: ( Env i local , π i , V i ) (Algorithm 3) end for end for

Original: Evaluate policies { π i } ni =1 on joint simulator Env joint end for
Revised: Evaluate policies { π i } ni =1 on joint simulator Env joint end for

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate the performance of CD-PPO in two IM problems: one contains 5 SKUs while the other contains 50 SKUs.
Revised: We evaluate the performance of CD-PPO in three IM problems, which contain 5, 50, and SKUs, respectively.

Original: By changing space size of the warehouse, we further testify how CD-PPO performs comparing to a series of baselines under different levels of competition.
Revised: By changing space size of the warehouse, we further testify how CD-PPO performs comparing to a series of baselines under different levels of competition.

Original: On these settings, we demonstrate that our algorithm can achieve comparable results as SOTA baselines, but is more sample-efficient.
Revised: On these settings, we demonstrate that our algorithm can achieve comparable results as SOTA baselines, but is more sample-efficient.

Original: 4. E XPERIMENT SETUPS
Revised: 4. E XPERIMENT SETUPS

Original: Our experiment is conducted on a simulator which can support both retailing and replenishment for multiple SKUs in one store.
Revised: Our experiment is conducted on a simulator which can support both retailing and replenishment for multiple SKUs in one store.

Original: Instead of sampling demands from some hypothetical distributions, we instead directly use authentic demand data from Kaggle (Makridakis et al., 2020), which contains sales history of five years (2011-2016) for various SKUs in Walmart.
Revised: Instead of sampling demands from some hypothetical distributions, we instead directly use authentic demand data from Kaggle (Makridakis et al., 2020), which contains sales history of five years (2011-2016) for various SKUs in Walmart.

Original: We randomly choose 55 SKUs from the data and use sales of the first four years as our training set, while the others are the testing set.
Revised: We randomly choose 155 SKUs from the data and use sales of the first four years as our training set, while the others are the testing set.

Original: For all other information, e.g. price, cost, leading time etc., that are necessary to instantiate our simulator but not included in the data set, we will randomly sample them from certain reasonable ranges.
Revised: For all other information, e.g. price, cost, leading time etc., that are necessary to instantiate our simulator but not included in the data set, we will randomly sample them from certain reasonable ranges.

Original: The evaluation metric is the total profit in dollars.
Revised: The evaluation metric is the total profit in dollars.

Original: For all results that we present, we run each of all algorithms for four times with random seeds and present the average performance with standard deviations.
Revised: For all results that we present, we run each of all algorithms for four times with random seeds and present the average performance with standard deviations.

Original: The source code as well as instructions to reproduce our results can be found in https://anonymous.4open.science/r/ replenishment-marl-baselines-75F4 .
Revised: The source code as well as instructions to reproduce our results can be found in https://anonymous.4open.science/r/ replenishment-marl-baselines-75F4 .

Original: Note that the simulator we are using is developed for general purposes, it contains extra details like replenishing orders fulfillment scheduling, transportation management etc.
Revised: Note that the simulator we are using is developed for general purposes, it contains extra details like replenishing orders fulfillment scheduling, transportation management etc.

Original: , hence it requires lots of computation resources to simulate IM problems with many SKUs ( > 100 ).
Revised: , hence it requires lots of computation resources to simulate IM problems with many SKUs ( > 100 ).

Original: This also hinder us to extend our experiments to cases with more than 100 SKUs on a single machine.
Revised: This also hinder us to extend our experiments to cases with more than 100 SKUs on a single machine.

Original: We leave it our future work to further testify our algorithm in a distributed environment where simulating large scale
Revised: We leave it our future work to further testify our algorithm in a distributed environment where simulating large scale

Original: IM problems is possible.
Revised: IM problems is possible.

Original: We compare our method CD-PPO with the strong model-free on-policy MARL algorithms
Revised: We compare our method CD-PPO with the strong model-free on-policy MARL algorithms

Original: MAPPO (Yu et al., 2021) and IPPO (de Witt et al., 2020) on 5-SKUs environment and we do not show the performance of COMA (Foerster et al., 2018) and Value Decomposition methods such as
Revised: MAPPO (Yu et al., 2021) and IPPO (de Witt et al., 2020) on 5-SKUs environment and we do not show the performance of COMA (Foerster et al., 2018) and Value Decomposition methods such as

Original: QMIX (Rashid et al., 2018) due to their bad performance (i.e. negative profit).
Revised: QMIX (Rashid et al., 2018) due to their bad performance (i.e. negative profit).

Original: As for 50-SKUs and 100-SKUs secenario, we can only run IPPO, since MAPPO fails quickly as it suffers from the huge joint state-action space for training a centralized critic.
Revised: As for 50-SKUs and 100-SKUs scenario, we can only run IPPO, since MAPPO fails quickly as it suffers from the huge joint state-action space for training a centralized critic.

Original: We also have added a well-known algorithm ”Base-stock” from OR community as our OR-based baseline (Corresponding introduction are shown in Appendix F).
Revised: Besides RL approaches, we also compare with base-stock policy, which is a well-known policy from OR community and widely adopted in practice (Kapuscinski & Tayur, 1999; Hubbs et al., 2020). (More details can be found in Appendix F).

Original: Our implementation is based on the EPyMARL (Papoudakis et al., 2021) framework, which contains MAPPO, IPPO and several common-used MARL algorithms.
Revised: Our implementation is based on the EPyMARL (Papoudakis et al., 2021) framework, which contains MAPPO, IPPO and several common-used MARL algorithms.

Original: As all agents are homogenous in IM, we let parameters of policy network and critic network be shared amongst all agents.
Revised: As all agents are homogeneous in IM, we let parameters of policy network and critic network be shared amongst all agents.

Original: The two networks are all constructed by a two-layer MLP with hidden size 64.
Revised: The two networks are all constructed by a two-layer MLP with hidden size 64.

Original: In our experiment, the actor network maps agent-specific state to a categorical distribution over a discrete action space { 0 , 13 , 23 , 1 , 43 , 53 , 2 , 52 , 3 , 4 , 5 , 6 , 7 , 9 , 12 } , such that the real replenishment quantity is obtained by multiplying the action with an agent’s sales mean of the past two weeks.
Revised: In our experiment, the actor network maps agent-specific state to a categorical distribution over a discrete action space { 0 , 13 , 23 , 1 , 43 , 53 , 2 , 52 , 3 , 4 , 5 , 6 , 7 , 9 , 12 } , such that the real replenishment quantity is obtained by multiplying the action with an agent’s sales mean of the past two weeks.

Original: In addition, to encourage policies to accommodate diverse context dynamics, we propose two methods to augment the context dynamics: 1) add noise to randomly chosen items with a predefined probability; 2) replace randomly chosen items with predicted values coming from a context generator model, which is also guided by the predefined probability.
Revised: In addition, to encourage policies to accommodate diverse context dynamics, we propose two methods to augment the context dynamics: 1) add noise to randomly chosen items with a predefined probability; 2) replace randomly chosen items with predicted values coming from a context generator model, which is also guided by the predefined probability.

Original: Besides, we also use the data collected in Algorithm 2 for policy training.
Revised: Besides, we also use the data collected in Algorithm 2 for policy training.

Original: More details about the algorithm can be found in
Revised: More details about the algorithm can be found in

Original: Appendix C.
Revised: Appendix C.

Original: 4. M AIN R ESULTS
Revised: 4. M AIN R ESULTS

Original: We evaluate CD-PPO, MAPPO and IPPO on 5,50,100-SKUs environments with different sizes of warehouse spaces.
Revised: We evaluate CD-PPO, MAPPO and IPPO on 5,50,100-SKUs environments with different sizes of warehouse spaces.

Original: IPPO and MAPPO only use individual rewards rather than team reward (summation of all individual rewards) to train critics.
Revised: IPPO and MAPPO only use individual rewards rather than team reward (summation of all individual rewards) to train critics.

Original: For a fair comparison, we also train IPPO and
Revised: For a fair comparison, we also train IPPO and

Original: MAPPO with information related to shared resource.
Revised: MAPPO with information related to shared resource.

Original: The training curves for 5-SKUs scenario areshown in Figure 2.
Revised: The training curves for 5-SKUs scenario are

Original: It is straightforward to observe that CD-PPO converges to the same performance

Original: [Table]
Revised: [Table]

Original:  comparing with other methods.
Revised: shown in Figure 2. It is straightforward to observe that CD-PPO converges to the same performance comparing with other methods.

Original: In particular, CD-PPO is more sample-efficient due to its local simulations in parallel.
Revised: In particular, CD-PPO is more sample-efficient due to its local simulations in parallel.

Original: To evaluate our algorithm on a larger scenario, we also run CD-PPO and IPPO (same with previous settings) on 50 and 100 SKUs and record the number of data samples when reaching the median performance of baselines.
Revised: To evaluate our algorithm on a larger scenario, we also run CD-PPO and IPPO (same with previous settings) on 50 and 100 SKUs and record the number of data samples when reaching the median performance of baselines.

Original: The results are summarized in Table 1 and Figure 3, where “N” and “C” denote the number of SKUs and the maximum capacity of shared resource, respectively.
Revised: The results are summarized in Table 1 and Figure 3, where “N” and “C” denote the number of SKUs and the maximum capacity of shared resource, respectively.

Original: More details about the implementation and hyperparameters we used can be found in
Revised: More details about the implementation and hyper-parameters we used can be found in

Original: Appendix C. for CD-PPO.
Revised: Appendix C. as parts of its inputs.

Revised: The X-axis † also takes in samples from local simulation for CD-PPO.

Original: As demonstrated in Figure 2, Figure 3 and Table 1, CD-PPO is able to produce results comparable to other strong baselines and even continues improving as the training proceeds.
Revised: As demonstrated in Figure 2, Figure 3 and Table 1, CD-PPO is able to produce results comparable to other strong baselines and even continues improving as the training proceeds.

Original: Moreover, our algorithm can curtail non-stationary effects as in centralized training and, in the meanwhile, can scale up to scenarios with a large number of agents.
Revised: Moreover, our algorithm can curtail non-stationary effects as in centralized training and, in the meanwhile, can scale up to scenarios with a large number of agents.

Original: In contrast, traditional MARL methods with CTDE paradigm even cannot start running on the 50-SKUs and the 100-SKUS environment, since the input of its critic is too huge to fit in the memory.
Revised: In contrast, traditional MARL methods with CTDE paradigm even cannot start running on the 50-SKUs and the 100-SKUS environment, since the input of its critic is too huge to fit in the memory.

Original: IPPO, on the other hand, can run successfully, but lack stability under different levels of warehouse spaces.
Revised: IPPO, on the other hand, can run successfully, but lack stability under different levels of warehouse spaces.

Original: The full results and more ablation studies about how the capacity of shared resource affects the performance of CD-PPO and the influence of augmentation for context dynamics can be found in Appendix G.
Revised: The full results and more ablation studies about how the capacity of shared resource affects the performance of CD-PPO and the influence of augmentation for context dynamics can be found in Appendix G.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: In this section we will introduce relevant prior work including studies of IM problem and common-used training paradigms in MARL.
Revised: In this section we will introduce relevant prior work including studies of IM problem and common-used training paradigms in MARL.

Original: More detailed description of other related work(e.g.
Revised: More detailed description of other related work(e.g.

Original: Constrained/Weakly-Coupled MDP, Model-Based MARL and Mean-Filed MARL) are shown in
Revised: Constrained/Weakly-Coupled MDP, Model-Based MARL and Mean-Filed MARL) are shown in

Original: Appendix A.
Revised: Appendix A.

Original: 5. I NVENTORY M ANAGEMENT
Revised: 5. I NVENTORY M ANAGEMENT

Original: Since the pioneer work in (Fukuda, 1964), many approaches have been proposed to solve different variants of IM problems, either using exact (Goldberg et al., 2016;
Revised: Since the pioneer work in (Fukuda, 1964), many approaches have been proposed to solve different variants of IM problems, either using exact (Goldberg et al., 2016;

Original: Huh et al., 2009) or approximate (Halman et al., 2009; Fang et al., 2013; Chen & Yang, 2019) dynamic programming.
Revised: Huh et al., 2009) or approximate (Halman et al., 2009; Fang et al., 2013; Chen & Yang, 2019) dynamic programming.

Original: As reinforcement learning based approaches are our main focus, we only present related work of this branch in the following.
Revised: As reinforcement learning based approaches are our main focus, we only present related work of this branch in the following.

Original: Interested readers will be referred to (Gijsbrechts et al., 2019) for an overview of traditional approaches for IM.
Revised: Interested readers will be referred to (Gijsbrechts et al., 2019) for an overview of traditional approaches for IM.

Original: The attempt to apply reinforcement learning to solve inventory management problem has a long history, see for instance (Giannoccaro & Pontrandolfo, 2002; Jiang & Sheng, 2009; Kara & Dogan, 2018; Barat et al., 2019; Gijsbrechts et al., 2019; Oroojlooyjadid et al., 2017; 2020).
Revised: The attempt to apply reinforcement learning to solve inventory management problem has a long history, see for instance (Giannoccaro & Pontrandolfo, 2002; Jiang & Sheng, 2009; Kara & Dogan, 2018; Barat et al., 2019; Gijsbrechts et al., 2019; Oroojlooyjadid et al., 2017; 2020).

Original: However, as their main focus is to deal with challenges like volatile customer demands, bullwhip effects etc.
Revised: However, as their main focus is to deal with challenges like volatile customer demands, bullwhip effects etc.

Original: in
Revised: in

Original: IM, they are restricted to simplified scenarios with only a single SKU.
Revised: IM, they are restricted to simplified scenarios with only a single SKU.

Original: While these approaches are able to outperform traditional approaches in these scenarios, they overlook system constraints and coordination of SKUs imposed by shared resources.
Revised: While these approaches are able to outperform traditional approaches in these scenarios, they overlook system constraints and coordination of SKUs imposed by shared resources.

Original: Exceptions are two recent works (Barat et al., 2019; Sultana et al., 2020), where more realistic scenarios containing multiple SKUs are considered.
Revised: Exceptions are two recent works (Barat et al., 2019; Sultana et al., 2020), where more realistic scenarios containing multiple SKUs are considered.

Original: In (Barat et al., 2019), the main contribution is to propose a framework that supports efficient deployment of RL algorithms in real systems.
Revised: In (Barat et al., 2019), the main contribution is to propose a framework that supports efficient deployment of RL algorithms in real systems.

Original: As an example, the authors also introduce a centralized algorithm for solving IM problems.
Revised: As an example, the authors also introduce a centralized algorithm for solving IM problems.

Original: In contrast, a decentralized algorithm is proposed in (Sultana et al., 2020) to solve IM problems with not only multiple SKUs but also multiple echelons.
Revised: In contrast, a decentralized algorithm is proposed in (Sultana et al., 2020) to solve IM problems with not only multiple SKUs but also multiple echelons.

Original: In both works, the training algorithm is the advantage actor critic (A2C)
Revised: In both works, the training algorithm is the advantage actor critic (A2C)

Original: (Wu et al., 2018).
Revised: (Wu et al., 2018).

Original: 5. T RAINING P ARADIGM IN MARL
Revised: 5. T RAINING P ARADIGM IN MARL

Original: MARL algorithms generally fall between two frameworks: centralized and decentralized learning.
Revised: MARL algorithms generally fall between two frameworks: centralized and decentralized learning.

Original: There are two lines of fully decentralized training: Independent Learning methods (IL)
Revised: There are two lines of fully decentralized training: Independent Learning methods (IL)

Original: (Tan, 1993; de Witt et al., 2020) and decentralized training with communication (Zhang et al., 2018;
Revised: (Tan, 1993; de Witt et al., 2020) and decentralized training with communication (Zhang et al., 2018;

Original: Sukhbaatar et al., 2016; Peng et al., 2017).
Revised: Sukhbaatar et al., 2016; Peng et al., 2017).

Original: For IL, each agent is learning independently to optimize its own reward and perceives the other agents as part of the environment.
Revised: For IL, each agent is learning independently to optimize its own reward and perceives the other agents as part of the environment.

Original: As for fully decentralized methods, representative methods usually build a direct communication pipe to share the message amongst agents to avoid non-stationary issue in MARL framework.
Revised: As for fully decentralized methods, representative methods usually build a direct communication pipe to share the message amongst agents to avoid non-stationary issue in MARL framework.

Original: One part of fully centralized approaches (Claus & Boutilier, 1998) assume a cooperative game and directly extend single-agent
Revised: One part of fully centralized approaches (Claus & Boutilier, 1998) assume a cooperative game and directly extend single-agent

Original: RL algorithms by learning a single policy to produce the joint actions of all agents simultaneously.
Revised: RL algorithms by learning a single policy to produce the joint actions of all agents simultaneously.

Original: Another type of centralized methods is called value decomposition(VD), which typically represents the joint Q-function as a function of agents’ local Q-functions (Sunehag et al., 2017; Son et al., 2019; Rashid et al., 2018) and has been considered a gold standard in MARL.
Revised: Another type of centralized methods is called value decomposition(VD), which typically represents the joint Q-function as a function of agents’ local Q-functions (Sunehag et al., 2017; Son et al., 2019; Rashid et al., 2018) and has been considered a gold standard in MARL.

Original: In contrast to previous methods, Centralised Training Decentralised Execution (CTDE) allows sharing of information during training, while policies are only conditioned on the agents’ local observations enabling decentralised execution.
Revised: In contrast to previous methods, Centralised Training Decentralised Execution (CTDE) allows sharing of information during training, while policies are only conditioned on the agents’ local observations enabling decentralised execution.

Original: The main category of CTDE algorithms are centralised policy gradient methods in which each agent consists of a decentralised actor and a centralised critic, which is optimised based on shared information between the agents.
Revised: The main category of CTDE algorithms are centralised policy gradient methods in which each agent consists of a decentralised actor and a centralised critic, which is optimised based on shared information between the agents.

Original: Representative studies of CTDE are
Revised: Representative studies of CTDE are

Original: MADDPG (Lowe et al., 2017), COMA (Foerster et al., 2018) and MAPPO (Yu et al., 2021) etc.
Revised: MADDPG (Lowe et al., 2017), COMA (Foerster et al., 2018) and MAPPO (Yu et al., 2021) etc.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we address inventory management problem for a single store with a large number of SKUs.
Revised: In this paper, we address inventory management problem for a single store with a large number of SKUs.

Original: Our method is based on shared resource structure, where agents only interact indirectly with each other through shared-resource.
Revised: Our method is based on shared resource structure, where agents only interact indirectly with each other through shared-resource.

Original: By leveraging such structure, our method can outperform existing MARL algorithms in terms of both final performance and computation efficiency.
Revised: By leveraging such structure, our method can outperform existing MARL algorithms in terms of both final performance and computation efficiency.

Original: It is worth mentioning that our method is not limited to IM, but also applicable to a wide range of real-world applications with shared-resource structure.
Revised: It is worth mentioning that our method is not limited to IM, but also applicable to a wide range of real-world applications with shared-resource structure.

Original: In real-world applications, we usually need to deal with thousands of agents, which poses a challenge for existing MARL algorithms.
Revised: In real-world applications, we usually need to deal with thousands of agents, which poses a challenge for existing MARL algorithms.

Original: To address such challenges, we need develop efficient algorithms which are compatible with distributed training.
Revised: To address such challenges, we need develop efficient algorithms which are compatible with distributed training.

Original: In this paper, we take our first step towards developing efficient and practical MARL algorithms for real-world applications with shared-resource structure, and will continue to address above challenges arisen in real-world applications in our future work.
Revised: In this paper, we take our first step towards developing efficient and practical MARL algorithms for real-world applications with shared-resource structure, and will continue to address above challenges arisen in real-world applications in our future work.

Original: A R ELATED W ORK A. T RAINING P ARADIGM IN MARL
Revised: A R ELATED W ORK A. T RAINING P ARADIGM IN MARL

Original: In this section we will provide a more detailed description of related work for training paradigm used in MARL.
Revised: In this section we will provide a more detailed description of related work for training paradigm used in MARL.

Original: From the perspective of training schemes, it can be devided into three categories: decentralized training decentralized execution (DTDE), centralized training centralized execution (CTCE) and centralized training decentralized execution (CTDE).
Revised: From the perspective of training schemes, it can be devided into three categories: decentralized training decentralized execution (DTDE), centralized training centralized execution (CTCE) and centralized training decentralized execution (CTDE).

Original: Recent deep MARL works often use the CTDE or CTCE training pradigm.
Revised: Recent deep MARL works often use the CTDE or CTCE training pradigm.

Original: The CTCE paradigm allows the straightforward employment of single-agent approaches such as Actor-Critic (Mnih et al., 2016) or policy gradient algorithms (Schulman et al., 2017) to multi-agent problems.
Revised: The CTCE paradigm allows the straightforward employment of single-agent approaches such as Actor-Critic (Mnih et al., 2016) or policy gradient algorithms (Schulman et al., 2017) to multi-agent problems.

Original: The representative work of CTCE is (Gupta et al., 2017), which represented the centralized executor as a set of independent sub-policies such that agents’ individual action distributions are captured rather than the joint action distribution of all agents.
Revised: The representative work of CTCE is (Gupta et al., 2017), which represented the centralized executor as a set of independent sub-policies such that agents’ individual action distributions are captured rather than the joint action distribution of all agents.

Original: Value-based CTDE approaches, which are also known as Value Decomposition methods (Peng et al., 2020; Mahajan et al., 2019; Rashid et al., 2020; 2018; Son et al., 2019; Sunehag et al., 2017;
Revised: Value-based CTDE approaches, which are also known as Value Decomposition methods (Peng et al., 2020; Mahajan et al., 2019; Rashid et al., 2020; 2018; Son et al., 2019; Sunehag et al., 2017;

Original: Wang et al., 2020c; 2019), mianly focus on how centrally learned value functions can be reasonably decoupled into decentralized ones and have shown promising results.
Revised: Wang et al., 2020b; 2019), mianly focus on how centrally learned value functions can be reasonably decoupled into decentralized ones and have shown promising results.

Original: Policy-gradient-based methods on CTDE, on the other hand, have heavily relied on centralized critics.
Revised: Policy-gradient-based methods on CTDE, on the other hand, have heavily relied on centralized critics.

Original: One of the first works utilizating a centralized critic was COMA (Foerster et al., 2018), a framework adopting a centralized critic with a counterfactual baseline.
Revised: One of the first works utilizating a centralized critic was COMA (Foerster et al., 2018), a framework adopting a centralized critic with a counterfactual baseline.

Original: For convergence properties, COMA establishes that the overall effect on decentralized policy gradient with a centralized critic can be reduced to a single-agent actor-critic approach, which ensures convergence under the similar assumptions like
Revised: For convergence properties, COMA establishes that the overall effect on decentralized policy gradient with a centralized critic can be reduced to a single-agent actor-critic approach, which ensures convergence under the similar assumptions like

Original: A2C.
Revised: A2C.

Original: Concurrently with COMA, MADDPG (Lowe et al., 2017) proposed to use a dedicated centralized critic for each agent in semi-competitive domains, demonstrating compelling empirical results in continuous action environments.
Revised: Concurrently with COMA, MADDPG (Lowe et al., 2017) proposed to use a dedicated centralized critic for each agent in semi-competitive domains, demonstrating compelling empirical results in continuous action environments.

Original: Recently, MAPPO (Yu et al., 2021), an on-policy policy gradient multi-agent reinforcement learning algorithm, achieves strong results comparable to the state-of-the-art on a variety of cooperative multi-agent challenges.
Revised: Recently, MAPPO (Yu et al., 2021), an on-policy policy gradient multi-agent reinforcement learning algorithm, achieves strong results comparable to the state-of-the-art on a variety of cooperative multi-agent challenges.

Original: Despite its on-policy nature,
Revised: Despite its on-policy nature,

Original: MAPPO is competitive to ubiquitous off-policy methods such as MADDPG, QMix, and RODE in terms of final performance, and in the vast majority of cases, comparable to off-policy methods in terms of sample-efficiency.
Revised: MAPPO is competitive to ubiquitous off-policy methods such as MADDPG, QMix, and RODE in terms of final performance, and in the vast majority of cases, comparable to off-policy methods in terms of sample-efficiency.

Original: In addition, many incremental research inspired by MADDPG, COMA or MAPPO also borrowed the centralized critic baselines e.g. M3DDPG (Li et al., 2019), SQDDPG (Wang et al., 2020a), etc.
Revised: In addition, many incremental research inspired by MADDPG, COMA or MAPPO also borrowed the centralized critic baselines e.g. M3DDPG (Li et al., 2019), SQDDPG (Wang et al., 2020a), etc.

Original: Mean Filed Q-Learning (Yang et al., 2018b; Carmona et al., 2019b) takes a different approach from the CTDE based methods.
Revised: Mean Filed Q-Learning (Yang et al., 2018; Carmona et al., 2019) takes a different approach from the CTDE based methods.

Original: It employs mean field approximation over the joint action space in order to address the scalability issue that exists in the prior methods.
Revised: It employs mean field approximation over the joint action space in order to address the scalability issue that exists in the prior methods.

Original: Contrary to CTDE, in DTDE paradigm, each agent has an associated policy which maps local observations to a distribution over individual actions.
Revised: Contrary to CTDE, in DTDE paradigm, each agent has an associated policy which maps local observations to a distribution over individual actions.

Original: No information is shared between agents such that each agent learns independently.
Revised: No information is shared between agents such that each agent learns independently.

Original: DTDE has been applied to cooperative navigation task (Chen et al., 2017; Strouse et al., 2018), to partially observable domains (Dobbe et al., 2017; Srinivasan et al., 2018), and to social dilemmas (Leibo et al., 2017).
Revised: DTDE has been applied to cooperative navigation task (Chen et al., 2017; Strouse et al., 2018), to partially observable domains (Dobbe et al., 2017; Srinivasan et al., 2018), and to social dilemmas (Leibo et al., 2017).

Original: For more comparisons of centralized critic and decentralized critic, please see (Lyu et al., 2021).
Revised: For more comparisons of centralized critic and decentralized critic, please see (Lyu et al., 2021).

Original: In this paper, we design a decentralized traning paradigm avoiding the flaws of traditional training paradigms proposed in literature.
Revised: In this paper, we design a decentralized traning paradigm avoiding the flaws of traditional training paradigms proposed in literature.

Original: The fundamental drawback of the DTDE paradigm is that the environment appears non-stationary from a single agent’s viewpoint because agents neither have access to the knowledge of others, nor do they perceive the joint action.
Revised: The fundamental drawback of the DTDE paradigm is that the environment appears non-stationary from a single agent’s viewpoint because agents neither have access to the knowledge of others, nor do they perceive the joint action.

Original: Some studies reported that
Revised: Some studies reported that

Original: DTDE scale poorly with the number of agent due to the extra sample complexity, which is added to the learning problem (Gupta et al., 2017);
Revised: DTDE scale poorly with the number of agent due to the extra sample complexity, which is added to the learning problem (Gupta et al., 2017);

Original: An obvious flaw for CTDE/CTCE is that state-action spaces grow exponentially by the number of agents.
Revised: An obvious flaw for CTDE/CTCE is that state-action spaces grow exponentially by the number of agents.

Original: Even though there are some attempts proposed that the joint model can be factored into individual policies for each agent To address the so-called curse of dimensionality, CTDE/CTCE methods have to use at least joint states overall agents as the input to approximate global value function to give guidance for centralized critics or decentralized policies.
Revised: Even though there are some attempts proposed that the joint model can be factored into individual policies for each agent To address the so-called curse of dimensionality, CTDE/CTCE methods have to use at least joint states overall agents as the input to approximate global value function to give guidance for centralized critics or decentralized policies.

Original: Meaning that these traditional training schemes still not have strong expansion capabilities to large number of agents when system’s state is combined by local state for each agent.
Revised: Meaning that these traditional training schemes still not have strong expansion capabilities to large number of agents when system’s state is combined by local state for each agent.

Original: As for our approach, we train agents independently with learned dynamics model of utilization’s trend for shared resource, which will give agent enough information to learn optimal policy (we will explain it in the following sections).
Revised: As for our approach, we train agents independently with learned dynamics model of utilization’s trend for shared resource, which will give agent enough information to learn optimal policy (we will explain it in the following sections).

Original: At the meantime, we can also, improve efficiency of data sampling since we don’t always use the original joint simulator containing all agents for data collection.
Revised: At the meantime, we can also, improve efficiency of data sampling since we don’t always use the original joint simulator containing all agents for data collection.

Original: Instead, we mainly running the light-cost local simulator by embedding the learned dynamic model, which can significantly reduce the cost of data collection process, especially when running joint simulator (such as inventory management) expensively.
Revised: Instead, we mainly running the light-cost local simulator by embedding the learned dynamic model, which can significantly reduce the cost of data collection process, especially when running joint simulator (such as inventory management) expensively.

Original: A. MDP S CENARIOS IN MARL
Revised: A. MDP S CENARIOS IN MARL

Original: There are limited similar variants of MDP settings which have been studied under MARL framework, to our knowledge.
Revised: There are limited similar variants of MDP settings which have been studied under MARL framework, to our knowledge.

Original: Dec-POMDP (Oliehoek & Amato, 2016) is the most common setting studied in MARL research, especially in fully cooperative tasks in which agents share the global team reward rather than individual rewards; In the scenarios of Constrained MDP setting (C-MDP, (Bhatnagar & Lakshmanan, 2012; Wang et al., 2019; Diddigi et al., 2019)), there are some constraints in the system, such as the penalty caused by illegal actions in autonomous driving (Zhang et al., 2016; Fowler et al., 2018), the resource-allocation constraints in scheduling tasks (Agrawal et al., 2016; Dolgov & Durfee, 2006), or limited bandwidth during communicating among agents (Fowler et al., 2018), etc.
Revised: Dec-POMDP (Oliehoek & Amato, 2016) is the most common setting studied in MARL research, especially in fully cooperative tasks in which agents share the global team reward rather than individual rewards; In the scenarios of Constrained MDP setting (C-MDP, (Bhatnagar & Lakshmanan, 2012; Wang et al., 2019; Diddigi et al., 2019)), there are some constraints in the system, such as the penalty caused by illegal actions in autonomous driving (Zhang et al., 2016; Fowler et al., 2018), the resource-allocation constraints in scheduling tasks (Agrawal et al., 2016; Dolgov & Durfee, 2006), or limited bandwidth during communicating among agents (Fowler et al., 2018), etc.

Original: Similar with C-MDP, Weakly-Coupled Constrained MDP (WC-C-MDP (Boutilier & Lu, 2016)) consider the problem of budget (or other resource) allocation in sequential decision problems involving a large number of concurrently running sub-processes,whose only interaction is through their consumption of budget.
Revised: Similar with C-MDP, Weakly-Coupled Constrained MDP (WC-C-MDP (Boutilier & Lu, 2016)) consider the problem of budget (or other resource) allocation in sequential decision problems involving a large number of concurrently running sub-processes,whose only interaction is through their consumption of budget.

Original: Different from mentioned scenarios, we focus on the situation where agents can only get the individual reward from the environment, so that team reward is invisible; Comparing with C-MDP and WC-C-MDP, the penalty of overstocking in which is related with stock levels of all agents in future, while in C-MDP and WC-C-MDP, the corresponding cost function is only based on historical states of the system.
Revised: Different from mentioned scenarios, we focus on the situation where agents can only get the individual reward from the environment, so that team reward is invisible; Comparing with C-MDP and WC-C-MDP, the penalty of overstocking in which is related with stock levels of all agents in future, while in C-MDP and WC-C-MDP, the corresponding cost function is only based on historical states of the system.

Original: Leading that we can not optimize only the combined objective of long-term summation of return and penalty.
Revised: Leading that we can not optimize only the combined objective of long-term summation of return and penalty.

Original: A. M ODEL -B ASED MARL
Revised: A. M ODEL -B ASED MARL

Original: For model-based MARL, there are relatively limited works of literature.
Revised: For model-based MARL, there are relatively limited works of literature.

Original: P-MARL (Marinescu et al., 2015) proposed an approach that integrates prediction and pattern change detection abilities into
Revised: P-MARL (Marinescu et al., 2015) proposed an approach that integrates prediction and pattern change detection abilities into

Original: MARL and thus minimises the effect of non-stationarity in the environment.
Revised: MARL and thus minimises the effect of non-stationarity in the environment.

Original: The environment is modelled as a time-series, with future estimates provided using prediction techniques.
Revised: The environment is modelled as a time-series, with future estimates provided using prediction techniques.

Original: Learning is based on the predicted environment behaviour, with agents employing this knowledge to improve their performance in real-time.
Revised: Learning is based on the predicted environment behaviour, with agents employing this knowledge to improve their performance in real-time.

Original: (Park et al., 2019) proposed to use a centralized auxiliary prediction network to model the environment dynamics to alleviate the non-stationary dynamics problem.
Revised: (Park et al., 2019) proposed to use a centralized auxiliary prediction network to model the environment dynamics to alleviate the non-stationary dynamics problem.

Original: (Krupnik et al., 2020) built a centralized multi-step generative model with a disentangled variational auto-encoder to predict the environment dynamics and the opponent actions and then performed trajectory planning.
Revised: (Krupnik et al., 2020) built a centralized multi-step generative model with a disentangled variational auto-encoder to predict the environment dynamics and the opponent actions and then performed trajectory planning.

Original: AORPO (Zhang et al., 2021) is a Dyna-style method in which each agent builds its multi-agent environment model that consist of a dynamics model and multiple opponent models, and trains its policy with the data both generated from adaptive opponent-wise rollout and interacted from the real environment.
Revised: AORPO (Zhang et al., 2021) is a Dyna-style method in which each agent builds its multi-agent environment model that consist of a dynamics model and multiple opponent models, and trains its policy with the data both generated from adaptive opponent-wise rollout and interacted from the real environment.

Original: To our knowledge, unlike previous work, our approach is the first to accelerate the learning process in MARL by building only a part dynamic model of the whole system.
Revised: To our knowledge, unlike previous work, our approach is the first to accelerate the learning process in MARL by building only a part dynamic model of the whole system.

Original: Our algorithm is also a Dyna-style method like AORPO.
Revised: Our algorithm is also a Dyna-style method like AORPO.

Original: And it is clear that the difficulty of modeling process will be reduced significantly and the learning process is more efficient.
Revised: And it is clear that the difficulty of modeling process will be reduced significantly and the learning process is more efficient.

Original: A. M EAN -F IELD MARL

Original: Existing literature has proposed various mean-field approximation approaches to model the population behavior of the agents for MARL with a large number.

Original: (Yang et al., 2018a) investigate a meanfield game with deterministic linear state transitions, and reformulate it as a mean-field MDP, where the mean-field state lies in finite-dimensional probability simplex.

Original: (Yang et al., 2018b) propose a mean-field approximation approach over actions, which approximates the interaction between any given agent and the population by the interaction between the agent’s action and the averaged actions of its neighboring agents.

Original: Such an averaging approach over the local actions, however, requires extensive prior knowledge.

Original: Several extended studies were proposed (Carmona et al., 2019a; Wang et al., 2020b; Li et al., 2021).

Original: The core idea of (Yang et al., 2018b) is to factorize the Q-function using only the pairwise local interactions and then used mean-field approximator Q i ( s, a i , a − i ) to estimate Q i ( s, a ) .

Original: In general, there are at least two differences between our approach and MFMARL methods:

Original: The key difference is that the interaction mechanism is not the same.

Original: MF-MARL assumes that agents can interact directly and only grasping the influence of neighbors is enough in the massive-agent setting.

Original: However, in IM problem with shared resources, there is no direct interaction between agents — agents can only exert limited indirect influence through the shared resource(i.e.

Original: , context).

Original: And item ”limited” means that from the individual agent’s view, the dynamics of the context is an aggregation of all other agents’ influences and it cannot be factorized to quantify how a specific agent affects the current agent.

Original: So that we believe that MF-MARL is not suitable for the IM problem since its assumption about how agents interact with each other does not hold for inventory management with shared resources.

Original: Due to the abstracted state in Stochastic Games, MF-MARL can scale up to the massiveagents setting since it addresses the exponential growth of action space.

Original: However, in the IM problem, the system state is the result of the concatenation of individual states of each agent with extra context.

Original: So that MF-MARL will also be suffered from this issue like other centralized MARL methods.

Original: And our CD-PPO avoids this issue by estimating the value function only with the individual state of each agent and the context.

Original: Specifically, the way of factorization is also very different.

Original: MF-MARL uses pairwise interaction Q i ( s, a i , a − i ) to approximate Q i ( s, a ) .

Original: On the contrary, CD-PPO uses Q i ( s i , a i , c ) as its approximation.

Original: We need to emphasize that, the core idea of MF

Original: -MARL is to use pairwise local interaction of actions as an approximation, while our method uses the context not the input actions to reflect the collective behavior of other agents.

Original:  B D ETAILS FOR I NVENTORY M ANAGEMENT P ROBLEM
Revised: A. M EAN -F IELD MARL B D ETAILS FOR I NVENTORY M ANAGEMENT P ROBLEM

Original: We summarize all notations for Section 2.2 in Table 2.
Revised: We summarize all notations for Section 2.2 in Table 2.

Original: We also give an example with 2 SKUs in
Revised: We also give an example with 2 SKUs in

Original: Fig.
Revised: Fig.

Original: 4 to further illustrate the whole procedure for inventory dynamics.
Revised: 4 to further illustrate the whole procedure for inventory dynamics.

Revised: [Table]

Original: C A LGORITHM D ETAILS
Revised: C A LGORITHM D ETAILS

Original: Here we provide the pseudocode of our algorithm with context augmentation in Algorithm 4.
Revised: Here we provide the pseudocode of our algorithm with context augmentation in Algorithm 4.

Original: And details of sub-algorithms will be introduced in the following subsections.
Revised: And details of sub-algorithms will be introduced in the following subsections.

Original: Notes that all pseudocode are assumed using RNN as networks.
Revised: Notes that all pseudocode are assumed using RNN as networks.

Original: [Table]
Revised: [Equation]

Original: C. G ET C ONTEXT D YNAMICS

Original: Our algorithm follows the algorithmic structure of the Multi-Agent PPO (Yu et al., 2021), in which we maintain two separate networks for π θ and V ϕ ( s ) .
Revised: Our algorithm follows the algorithmic structure of the Multi-Agent PPO (Yu et al., 2021), in which we maintain two separate networks for π θ and V ϕ ( s ) .

Original: And the first stage of our algorithm is only running policies in the origin joint environment to get episodes for context dynamics,
Revised: And the first stage of our algorithm is only running policies in the origin joint environment to get episodes for context dynamics,

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: e. the trajectories of c
Revised: e. the trajectories of c

Original: it and c − i t .
Revised: it and c − i t .

Original: This process is similar with all other traditional MARL methods.
Revised: This process is similar with all other traditional MARL methods.

Original: Notes that we can also save the transitions of each agent for learning policy and value function network.
Revised: Notes that we can also save the transitions of each agent for learning policy and value function network.

Original: In the following pseudocode of joint sampling(Algorithm 2), we only record the part for the context dynamics to train the context model prepared for next stage’s training [Table]
Revised: In the following pseudocode of joint sampling(Algorithm 2), we only record the part for the context dynamics to train the context model prepared for next stage’s training.

Original: After data collecting for context dynamics [Table] we will use a LSTM model f c to train a surrogate predictor as an extra augmentation for collected dynamics in next stage.
Revised: After data collecting for context dynamics, we will use a LSTM model f c to train a surrogate predictor as an extra augmentation for collected dynamics in next stage.

Original: In details [Table] we split the collected trajectories of context into several sub-sequences of length 8 and the training objective is to minimize the mean-squared error of the L + 1 day’s capacity predicted given the dynamics of previous
Revised: In details, we split the collected trajectories of context into several sub-sequences of length 8 and the training objective is to minimize the mean-squared error of the L + 1 day’s capacity predicted given the dynamics of previous

Original: L days.
Revised: L days.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  D ECENTRALIZED PPO
Revised: C. D ECENTRALIZED PPO

Original: With the collected context dynamics of the shared resource, it is easy to run the second stage: sampling on the local simulators for each agent and then training the policy and critic with data.
Revised: With the collected context dynamics of the shared resource, it is easy to run the second stage: sampling on the local simulators for each agent and then training the policy and critic with data.

Original: It is worth noting that the main difference between our training paradigm and traditional MARL methods under CTDE structure is that we directly sampling local observations in the extra simulators in which only one agent existed rather than the joint simulator in which all agents interacting with each other.
Revised: It is worth noting that the main difference between our training paradigm and traditional MARL methods under CTDE structure is that we directly sampling local observations in the extra simulators in which only one agent existed rather than the joint simulator in which all agents interacting with each other.

Original: In other words, in the new local simulator, there is only one SKU in the entire store, and the trend of available capacity is completely simulated according to the given context dynamics.
Revised: In other words, in the new local simulator, there is only one SKU in the entire store, and the trend of available capacity is completely simulated according to the given context dynamics.

Original: In practice, we parallelly initialize new instances of the original inventory environment with the new configure settings which only contains a specific SKU i and a fixed trajectory of context.
Revised: In practice, we parallelly initialize new instances of the original inventory environment with the new configure settings which only contains a specific SKU i and a fixed trajectory of context.

Original: As for the fixed context trajectory, we use the subtraction results { c − i } Tt =0 with some augmentations: 1) add some noise in some items with a predefined probability;2) replace some items with predicted values comes from the trained context model also by the predefined probability.
Revised: As for the fixed context trajectory, we use the subtraction results { c − i } Tt =0 with some augmentations: 1) add some noise in some items with a predefined probability;2) replace some items with predicted values comes from the trained context model also by the predefined probability.

Original: Then we run the policy to interact with the local simulators to conduct episodes under the embedded context dynamics and put them into a shared replay buffer since all transitions are homogeneous and we shared the parameters over all policies.
Revised: Then we run the policy to interact with the local simulators to conduct episodes under the embedded context dynamics and put them into a shared replay buffer since all transitions are homogeneous and we shared the parameters over all policies.

Original: And decentralized training will be started by utilizing the shared replay buffer of transitions collected from local simulators.
Revised: And decentralized training will be started by utilizing the shared replay buffer of transitions collected from local simulators.

Original: We consider a variant of the advantage function based on decentralized learning, where each agent learns a agent-specific state based critic V ϕ
Revised: We consider a variant of the advantage function based on decentralized learning, where each agent learns a agent-specific state based critic V ϕ

Original: ( s it ) parameterised by ϕ using Generalized Advantage
Revised: ( s it ) parameterised by ϕ using Generalized Advantage

Original: Estimation (GAE, (Schulman et al., 2016)) with discount factor γ = 0 .
Revised: Estimation (GAE, (Schulman et al., 2016)) with discount factor γ = 0 .

Original: 99 and λ = 0 .
Revised: 99 and λ = 0 .

Original: We also add an entropy regularization term to the final policy loss (Mnih et al., 2016).
Revised: We also add an entropy regularization term to the final policy loss (Mnih et al., 2016).

Original: For each agent i , we have its advantage estimation as follows:
Revised: For each agent i , we have its advantage estimation as follows:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where δ it i =
Revised: where δ it i =

Original: r t (cid:0) s it , a it (cid:1) + γV ϕ (cid:0)
Revised: r t (cid:0) s it , a it (cid:1) + γV ϕ (cid:0)

Original: z it +1 (cid:1) − V ϕ
Revised: z it + [Equation] − V ϕ

Original: (cid:0) s it (cid:1) is the TD error at time step t and h is marked as steps num.
Revised: [Equation] s it (cid:1) is the TD error at time step t and h is marked as steps num.

Original: And we use individual reward provided from local simulator r it ( s it , a it ) .
Revised: And we use individual reward provided from local simulator r it ( s it , a it ) .

Original: So that the final policy loss for each agent i becomes:
Revised: So that the final policy loss for each agent i becomes:

Original: [Equation]
Revised: [Equation]

Original: As for training value function, in addition to clipping the policy updates, our method also use value clipping to restrict the update of critic function for each agent i to be smaller than ϵ as proposed by
Revised: As for training value function, in addition to clipping the policy updates, our method also use value clipping to restrict the update of critic function for each agent i to be smaller than ϵ as proposed by

Original: GAE using:
Revised: GAE using:

Original: [Equation]
Revised: [Equation]

Original: where ϕ old are old parameters before the update and ˆ V it
Revised: where ϕ old are old parameters before the update and ˆ V it

Original: =
Revised: =

Original: A it + V ϕ (cid:0) s it (cid:1) .
Revised: A it + V ϕ (cid:0) s it (cid:1) .

Original: The update equation restricts the update of the value function to within the trust region, and therefore helps us to avoid overfitting to the most recent batch of data.
Revised: The update equation restricts the update of the value function to within the trust region, and therefore helps us to avoid overfitting to the most recent batch of data.

Original: For each agent, the overall learning loss becomes:
Revised: For each agent, the overall learning loss becomes:

Original: [Equation]
Revised: [Equation]

Original: It is obvious that all networks are trained in the decentralized way since their inputs are all local variables which stem from the light-cost local simulators.
Revised: It is obvious that all networks are trained in the decentralized way since their inputs are all local variables which stem from the light-cost local simulators.

Original: As mentioned before, at this learning stage, there are no interactions between any two agents.
Revised: As mentioned before, at this learning stage, there are no interactions between any two agents.

Original: Although it seems like the way of independent learning, we need to point that we use the global context simulated from the joint environment, which is essentially different from independent learning methods since they will not consider this style global information which is simulated from joint simulator but be fixed in the local simulators.
Revised: Although it seems like the way of independent learning, we need to point that we use the global context simulated from the joint environment, which is essentially different from independent learning methods since they will not consider this style global information which is simulated from joint simulator but be fixed in the local simulators.

Original: Our decentralized training have several advantages: firstly, the local simulator is running efficient because of its simple only-one-agent transition function; secondly, this paradigm avoid the issue for non-stationary occurred in the traditional MARL methods since there are no interaction amongst agents so that it is no need to consider influences of other agents; thirdly, we can use more diverse context trajectories to make agents face various situations of the available levels of the store, which leads to improve the generalization of the networks to be trained; fourthly, it is easy for this training paradigm to be extended to large-scale distributed training by running parallel simulation whose communication cost is also acceptable for modern distributed training frameworks.
Revised: Our decentralized training have several advantages: firstly, the local simulator is running efficient because of its simple only-one-agent transition function; secondly, this paradigm avoid the issue for non-stationary occurred in the traditional MARL methods since there are no interaction amongst agents so that it is no need to consider influences of other agents; thirdly, we can use more diverse context trajectories to make agents face various situations of the available levels of the store, which leads to improve the generalization of the networks to be trained; fourthly, it is easy for this training paradigm to be extended to large-scale distributed training by running parallel simulation whose communication cost is also acceptable for modern distributed training frameworks.

Original: If the critic and actor networks are RNNs, then the loss functions additionally sum over time, and the networks are trained via Backpropagation Through Time (BPTT).
Revised: If the critic and actor networks are RNNs, then the loss functions additionally sum over time, and the networks are trained via Backpropagation Through Time (BPTT).

Original: Pseudocode for local sampling with recurrent version of policy networks is shown in Algorithm 3.
Revised: Pseudocode for local sampling with recurrent version of policy networks is shown in Algorithm 3.

Original: Our algorithm follows a decentralized training paradigm, which is compatible with distributed training framework. We can run parallel local simulation while communication cost of context dynamics is also acceptable. With distributed training, our method can be much more efficient than learning from the joint simulator directly.
Revised: Algorithm

Original: INPUT policies { π iθ i } ni =1 and the joint simulator Env joint (Optional) Initialize ω , the parameters for context model
Revised: INPUT policies { π iθ i } ni =1 and the joint simulator Env joint (Optional) Initialize ω , the parameters for context model

Original: f c set data buffer D = {} for bs = 1 to batch size do τ = [] empty list initialize h 10 ,π , · · · h n 0 ,π actor RNN states for t = 1 to T do for all agents i do p it , h it,π = π
Revised: f c set data buffer D = {} for bs = 1 to batch size do τ = [] empty list initialize h 10 ,π , · · · h n 0 ,π actor RNN states for t = 1 to T do for all agents i do p it , h it,π = π

Original: i
Revised: i

Original: ( s it , c t , h it − 1 ,π ; θ i ) a it ∼ p it end for
Revised: ( s it , c t , h it − 1 ,π ; θ i ) a it ∼ p it end for

Original: Execute actions a t , observe r t , s t +1 τ +=
Revised: Execute actions a t , observe r t , s t +1 τ +=

Original: [ s t , c t , h t,π , a t , r t , s t +1 ,
Revised: [ s t , c t , h t,π , a t , r t , s t +1 ,

Original: c t +1 ] end for //
Revised: c t +1 ] end for //

Original: Split trajectory τ into chunks of length L for l = 0, 1, ..,
Revised: Split trajectory τ into chunks of length L for l = 0, 1, ..,

Original: T//L do
Revised: T//L do

Original: D = D ∪ ( c [ l : l + T ]) end for end for // (Optional)
Revised: D L OCAL S AMPLING V . S . J OINT S AMPLING

Original: Train the context model for augmentation if Need to train context model then for mini-batch k = 1 , . . , K 1 do b ← random mini-batch from D with all agent data Update capacity-context dynamics model with Eq.end for
Revised: Our algorithm follows a decentralized training paradigm, which is compatible with distributed training framework. We can run parallel local simulation while communication cost of context dynamics is also acceptable.

Original: Adam update ω with data b end if
Revised: With distributed training, our method can be much more efficient than learning from the joint simulator directly. One may argue that we can also learn from joint simulator efficiently by implementing a distributed/parallelized joint simulator. While this can indeed improve sampling efficiency in many scenarios, its usefulness is limited particularly for systems with a large number of agents.

Original: OUTPUT context dynamics { c − i t } Tt =1 for i = 1 , . . . , n ; (Optional) f ω
Revised: In a typ-

Original: INPUT local simulator Env ilocal , policy π iθ i and value function V iϕ i
Revised: INPUT local simulator Env ilocal , policy π iθ i and value function V iϕ i

Original: Set data buffer D = {}
Revised: Set data buffer D = {}

Original: Initialize h 10 ,π , · · · , h n 0 ,π actor RNN states
Revised: Initialize h 10 ,π , · · · , h n 0 ,π actor RNN states

Original: Initialize h 10 ,V , . . .
Revised: Initialize h 10 ,V , . . .

Original: h n 0 ,V critic RNN states τ =
Revised: h n 0 ,V critic RNN states τ =

Original: [] empty list for t = 1 to T do p it , h it,π = π
Revised: [] empty list for t = 1 to T do p it , h it,π = π

Original: i ( s it , h it − 1 ,π , c it ; θ i ) a it ∼ p it v it , h it,V = V i
Revised: i ( s it , h it − 1 ,π , c it ; θ i ) a it ∼ p it v it , h it,V = V i

Original: ( s it , c
Revised: ( s it , c

Original: it , h it − 1 ,V ; ϕ i )
Revised: it , h it − 1 ,V ; ϕ i )

Original: Execute actions a it in Env ilocal , and then observe r it ,
Revised: Execute actions a it in Env ilocal , and then observe r it ,

Original: c it +1 , s it + τ + =
Revised: c it +1 , s it + τ + =

Original: [ s it , c
Revised: [ s it , c

Original: it , a it , h it,π , h it,V , s it +1 , c
Revised: it , a it , h it,π , h it,V , s it +1 , c

Original: it +1 ] end for
Revised: it +1 ] end for

Original: Compute advantage estimate ˆ A via GAE on τ (Eq. 15)
Revised: Compute advantage estimate ˆ A via GAE on τ (Eq. 15)

Original: Compute reward-to-go ˆ R on τ // Split trajectory τ into chunks of length L in D for l = 0, 1, ..,
Revised: Compute reward-to-go ˆ R on τ // Split trajectory τ into chunks of length L in D for l = 0, 1, ..,

Original: T//L do
Revised: T//L do

Original: D
Revised: D

Original: = D ∪ ( τ [ l : l + T, ˆ A
Revised: = D ∪ ( τ [ l : l + T, ˆ A

Original: [ l : l + L ] , ˆ R [ l : l + L ]) end for for mini-batch k = 1 , . .
Revised: [ l : l + L ] , ˆ R [ l : l + L ]) end for for mini-batch k = 1 , . .

Original: , K 2 do b ← random mini-batch from D with all agent data for each data chunk c in the mini-batch b do update RNN hidden states for π i and V i from first hidden state in data chunk end for end for
Revised: , K 2 do b ← random mini-batch from D with all agent data for each data chunk c in the mini-batch b do update RNN hidden states for π i and V i from first hidden state in data chunk end for end for

Original: Calculate the overall loss according to Eq.
Revised: Calculate the overall loss according to Eq.

Original: 16 to Eq.
Revised: 16 to Eq.

Original: Adam update θ i on L i ( θ i ) and H with data b
Revised: Adam update θ i on L i ( θ i ) and H with data b

Original: Adam update ϕ
Revised: Adam update ϕ

Original: i on L i ( ϕ i ) with data b
Revised: i on L i ( ϕ i ) with data b

Original: OUTPUT policy π iθ i and value function V iϕ i value functions
Revised: OUTPUT policy π iθ i and value function V iϕ i value functions

Original: V i for i = 1 , . . . , n
Revised: V i for i = 1 , . . . , n

Original: Initialize context model f ω and the augmentation probability p aug for M epochs do // Collect context dynamics via running joint simulation { c − 1 t } Tt =0 , . . .
Revised: Initialize context model f ω and the augmentation probability p aug for M epochs do // Collect context dynamics via running joint simulation { c − 1 t } Tt =0 , . . .

Original: , { c − n t } Tt =0 , f ω ← GetContextDynamics ( Env joint , { π i } ni =1 , f ω ) (Algorithm 2) for k = 1 , 2 . .
Revised: , { c − n t } Tt =0 , f ω ← GetContextDynamics ( Env joint , { π i } ni =1 , f ω ) (Algorithm 2) for k = 1 , 2 . .

Original: , K do for all agents i do //
Revised: , K do for all agents i do //

Original: Set capacity trajectory by augmented context dynamics
Revised: Set capacity trajectory by augmented context dynamics

Original: Env ilocal .
Revised: Env ilocal .

Original: set c trajectory ( aug ( { c − i t } Tt =0 , f ω , p aug )) //
Revised: set c trajectory ( aug ( { c − i t } Tt =0 , f ω , p aug )) //

Original: Train policy by running simulation in the corresponding local environment π i , V
Revised: Train policy by running simulation in the corresponding local environment π i , V

Original: i ← DecentralizedPPO
Revised: i ← DecentralizedPPO

Original: ( Env i local , π i , V i )
Revised: ( Env i local , π i , V i )

Original: (Algorithm 3) end for end for
Revised: (Algorithm 3) end for end for

Original: Evaluate policies { π i } ni =1 on joint simulator Env joint end for One may argue that we can also learn from joint simulator efficiently by implementing a distributed/parallelized joint simulator. While this can indeed improve sampling efficiency in many scenarios, its usefulness is limited particularly for systems with a large number of agents. In a typical multi-agent system, interactions occur frequently among all agents.
Revised: Evaluate policies { π i } ni =1 on joint simulator Env joint end for ical multi-agent system, interactions occur frequently among all agents.

Original: Actually, one advantage of multi-agent systems is to model complex interactions among agents.
Revised: Actually, one advantage of multi-agent systems is to model complex interactions among agents.

Original: However, such interactions will remarkably reduce the efficiency improvement brought by parallelism, as these interactions often require involved agents to synchronize which usually consumes lots of time on waiting and communicating.
Revised: However, such interactions will remarkably reduce the efficiency improvement brought by parallelism, as these interactions often require involved agents to synchronize which usually consumes lots of time on waiting and communicating.

Original: For instance, in IM problems, at each time step, all agents should be synchronized in order to calculate ρ in Eq.
Revised: For instance, in IM problems, at each time step, all agents should be synchronized in order to calculate ρ in Eq.

Original: 6 before moving to the next time step.
Revised: 6 before moving to the next time step.

Original: On the other hand, in our local sampling approach we simplify such costly interactions by utilizing special structures of shared-resource stochastic games.
Revised: On the other hand, in our local sampling approach we simplify such costly interactions by utilizing special structures of shared-resource stochastic games.

Original: Under our approach, there is no need for agents to be synchronized before moving to the next time step in local simulators.
Revised: Under our approach, there is no need for agents to be synchronized before moving to the next time step in local simulators.

Original: As a result, our method can be much more efficient than learning only from the joint simulator in practice.
Revised: As a result, our method can be much more efficient than learning only from the joint simulator in practice.

Original: E T RAINING D ETAILS E. T HE C ODEBASE
Revised: E T RAINING D ETAILS E. T HE C ODEBASE

Original: As part of this work we extended the well-known EPyMARL codebase((Papoudakis et al., 2021)) to integrated our simulator and algorithm, which already include several common-used algorithms and support more environments as well as allow for more flexible tuning of the implementation details.
Revised: As part of this work we extended the well-known EPyMARL codebase((Papoudakis et al., 2021)) to integrated our simulator and algorithm, which already include several common-used algorithms and support more environments as well as allow for more flexible tuning of the implementation details.

Original: It is convenience for us to compare our algorithm with other baselines.
Revised: It is convenience for us to compare our algorithm with other baselines.

Original: All code for our new codebase is publicly available open-source on Anonymous GitHub under the following link: https:// anonymous.4open.science/r/replenishment-marl-baselines-75F
Revised: All code for our new codebase is publicly available open-source on Anonymous GitHub under the following link: https:// anonymous.4open.science/r/replenishment-marl-baselines-75F

Original: E. H YPERPARAMETERS D ETAILS
Revised: E. H YPERPARAMETERS D ETAILS

Original: Table 3 presents the hyperparameters used in our algorithm for 5-SKUs environment.
Revised: Table 3 presents the hyperparameters used in our algorithm for 5-SKUs environment.

Original: [Table]
Revised: [Table]

Original: E. D ETAILS OF S TATES AND R EWARDS
Revised: E. D ETAILS OF S TATES AND R EWARDS

Original: Table 4 shows the features of the state for our MARL agent that corresponds to the i -th SKU on the t -th time step.
Revised: Table 4 shows the features of the state for our MARL agent that corresponds to the i -th SKU on the t -th time step.

Original: It’s worthy to note that we use the profit generated on the i -th SKU at the t -th time step Pt it devided by 1000000 as the individual reward of i -th agent at the t -th time step, for team reward methods, we simply sum up all the individual rewards, which corresponds to the daily profit of the whole store at the t -th time step, devided by 1000000.
Revised: It’s worthy to note that we use the profit generated on the i -th SKU at the t -th time step Pt it divided by 1000000 as the individual reward of i -th agent at the t -th time step, for team reward methods, we simply sum up all the individual rewards, which corresponds to the daily profit of the whole store at the t -th time step, divided by 1000000.

Original: [Table]
Revised: [Table]

Original: F B ASE - STOCK P OLICY
Revised: F B ASE - STOCK P OLICY

Revised: Input: { D it } t 2 t = t 1 , v , τ , IM parameters

Original: In addition to comparing with related MARL baselines
Revised: Output: Z i , { O it } t 4 t = t// Description:Base-stock policy for single SKU i Z i :Base-stock level for SKU i //

Revised: { O it } t 4 t = t 3 :Base-stock replenishment policy for SKU i, from t 3 to t// { D it } t 2 t = t 1 :Demand series of SKU i used to infer Z i , from t 1 to t// v : v ≥ 1 , v ∈ R , Hyper-parameter to control storage utilization level // τ : τ ∈ N + , Hyper-parameter to control replenishing interval //

Revised: IM parameters:including leading time L i , storage capacity C , etc // Solve Mixed Integer Programming with dual simplex methods:

Revised: Z i ← max Z i (cid:88) t 2 t =

Original: {+t 
+}
Revised: {+t 
+}

Revised: Pt it s.t.

Revised:     

Revised: S it =

Revised: min (cid:0) D it , I it (cid:1)

Original: [Table]{+T it +
 = T it − O it − L+}
Revised: {+T it +
 = T it − O it − L+}

Original: {+i +
+}
Revised: {+i +
+}

Revised: + O it +

Revised: I i t

Revised: +1 =

Revised: I it − S

Revised: it + O it − L

Revised: i +

Revised: O it = max (cid:0) 0 , Z i − I

Revised: it − T

Revised: it (cid:1)

Revised: Pt it = p i S it − q

Revised: i

Revised: O it − hI

Revised: it t 1 ≤ t ≤ t// Replenishing policy deduction:

Revised: O it = max (cid:0) 0 , Z i − I

Revised: it − T

Revised: it (cid:1) , t 3 ≤ t ≤ t

Original: {+O it = min (cid:16) O it , vC − (cid:80) nj =
+}
Revised: {+O it = min (cid:16) O it , vC − (cid:80) nj =
+}

Revised: ( I jt + T jt ) (cid:17) , t 3 ≤ t ≤ t

Revised: O it

Original:  we also add a well-known algorithm ”Basestock” from OR community as our non-RL baseline.
Revised: = O it I [ t mod τ = 0] , t 3 ≤ t ≤ treturn Z i , { O it } t 4 t = t In addition to comparing with related MARL baselines, we also add a well-known algorithm ”Basestock” from OR community as our non-RL baseline.

Revised: The pseudo-code for base-stock policy can be found in Algorithm 5, where Z i is called the base-stock for agent i and is computed by solving a mixed integer programming problem.

Revised: After that, Z i will be used to guide replenishment of agent i periodically.

Revised: We shall note that base-stock policy can not deal with complex IM variant settings like coordinated replenishment (multiple SKUs with storage capacities), order cost and stochastic

Revised: VLTs, etc.

Revised: These complicated realistic constraints are exactly what we use to test other MARL algorithms.

Original:  Base-stock utilizes Linear Programming to work out the base stock levels and thus the replenishing policy. ”
Revised: Thus it may happen that Base-stock policies constantly overflow warehouse capacities when storage is tight, in which cases incoming products are abandoned proportionally as we explained in Section 2.2. This explains Base-stock’s poor performance on some of the envs. Base-stock utilizes Linear Programming to work out the base stock levels and thus the replenishing policy. ”

Original: Static” indicates that the base stock levels are based on demand data from training set and then kept static while making desicions on test set. ”
Revised: Static” indicates that the base stock levels are based on demand data from training set and then kept static while making decisions on test set. ”

Original: Dynamic” updates its base stock levels on a regular time cycle. ”
Revised: Dynamic” updates its base stock levels on a regular time cycle. ”

Original: Oracle” directly access the whole test set to calculate its base stock levels, which is essentially a cheating version used to show the upper limits of Base-stock policy.
Revised: Oracle” directly access the whole test set to calculate its base stock levels, which is essentially a cheating version used to show the upper limits of Base-stock policy.

Revised: In practice, we conduct grid search to find proper v and τ .

Original: We refer readers to (Hubbs et al., 2020) for a detailed description.
Revised: We refer readers to (Hubbs et al., 2020) for a detailed description.

Original: Our implementation of Base-stock baselines is also inspired by it.
Revised: Our implementation of Base-stock baselines is also inspired by it.

Original: G A DDITIONAL R ESULTS G. T HE F ULL R ESULTS ON A LL E NVIRONMENTS
Revised: G A DDITIONAL R ESULTS G. T HE F ULL R ESULTS ON A LL E NVIRONMENTS

Original: Here we present all the experiment results in Table 5 and display the training curves of algorithms in 50-SKUs scenarios in Figure 5 and Figure 6, we also present the results from OR methods designed to deal with IM problem, namely Base-stock, in Table [Table]
Revised: Here we present all the experiment results in Table 5 and display the training curves of algorithms in 50-SKUs scenarios in Figure 5 and Figure 6. We also present the results from OR methods designed to deal with IM problem, namely Base-stock, in Table 5.

Original: Please note that, actually, VLT(i.e leading time) for each SKU in N50 and N100 is stochastic during simulation.
Revised: Please note that, actually, VLT (i.e leading time) for each SKU in N50 and N100 is stochastic during simulation.

Original: On the one hand, the specified VLTs can be modelled as exponential distribution in our simulator.
Revised: On the one hand, the specified VLTs can be modelled as exponential distributions in our simulator.

Original: On the other hand, the real lead time for each procurement can be longer than specified due to lack of upstream vehicles.
Revised: On the other hand, the real lead time for each procurement can be longer than specified due to lack of upstream vehicles.

Original: (Distribution capacity is also considered in the simulator, though not the focus of this paper).
Revised: (Distribution capacity is also considered in the simulator, though not the focus of this paper).

Original: So that the results running on N50 and N100 are all considering stochastic VLTs
Revised: So that the results running on N50 and N100 are all considering stochastic VLTs.

Original: [Table]
Revised: [Table]

Original:  It’s worthy noting that throughout all the environments we’ve experimented on, CD-PPO enjoys higher sample efficiency as shown in Table 6.
Revised: [Table] It’s worthy noting that throughout all the environments we’ve experimented on [Table] CD-PPO enjoys higher sample efficiency as shown in Table 6.

Original: Though it seems that in environment where the storage capacity is extremely tense, like N5-C50, CD-PPO performs not as well as IPPO, we reason that these tense situations favor IPPO(with team reward), which may quickly learn how to adjust each agents’ behavior to improve team reward. While CD-PPO, owing to its decentralized training
Revised: Though it seems that in environment where the storage capacity is extremely tense, like N5-C50, CD-PPO performs not as well as IPPO, we reason that these tense situations favor IPPO (with team reward)

Original: [Table]
Revised: [Table]

Revised: which may quickly learn how to adjust each agents’ behavior to improve team reward.

Original:  paradigm, struggles to learn a highly cooperative policy to cope with the strong resource limits while meeting stochastic customer demands.
Revised: While CD-PPO, owing to its decentralized training paradigm, struggles to learn a highly cooperative policy to cope with the strong resource limits while meeting stochastic customer demands.

Original: Yet we still claim that CD-PPO does outperform its IR(w/o context) baselines while producing comparable performance when compared to IR(with context) algorithms.
Revised: Yet we still claim that CD-PPO does outperform its IR (w/o context) baselines while producing comparable performance when compared to IR (with context) algorithms.

Original: On the other hand, IPPO(with team reward) struggles to learn a good policy in 50agents environment for the large action space compared to the single team reward signal, and it fails when faced with N50-C2000 whose state space is even larger.
Revised: On the other hand, IPPO(with team reward) struggles to learn a good policy in 50-agents environment for the large action space compared to the single team reward signal, and it fails when faced with N50-C2000 whose state space is even larger.

Original: To top it all off, CD-PPO does have its strengths in terms of sample efficiency.
Revised: To top it all off, CD-PPO does have its strengths in terms of sample efficiency.

Original: Though we will continue to address the above challenge in our future work.
Revised: Though we will continue to address the above challenge in our future work.

Original: [Figure]

Original: G. A BLATION S TUDIES FOR C ONTEXT A UGMENTATION
Revised: G. A BLATION S TUDIES FOR C ONTEXT A UGMENTATION

Original: In this section, we aim to seek a better way to augment the context dynamics in order to boost performance of our algorithm.
Revised: In this section, we aim to seek a better way to augment the context dynamics in order to boost performance of our algorithm.

Original: We ponder the following two questions:
Revised: We ponder the following two questions:

Original: Q1: Which is a better way to augment the original data, adding noise or using a Deep prediction model?
Revised: Q1: Which is a better way to augment the original data, adding noise or using a Deep prediction model?

Original: To answer this question, we set out experiments on environment N5-C100 with our algorithm CDPPO, using either context trajectories generated by a deep LSTM prediction model, or simply adding a random normal perturbation to the original dynamics trajectories, both on 3 different seeds.
Revised: To answer this question, we set out experiments on environment N5-C100 with our algorithm CDPPO, using either context trajectories generated by a deep LSTM prediction model, or simply adding a random normal perturbation to the original dynamics trajectories, both on 3 different seeds.

Original: As shown in Figure 7, those runs with deep prediction model generated dynamics enjoy less std and better final performance.
Revised: As shown in Figure 7, those runs with deep prediction model generated dynamics enjoy less std and better final performance.

Original: This could result from that the diversity of deep model generated trajectories surpasses that of random perturbation.
Revised: This could result from that the diversity of deep model generated trajectories surpasses that of random perturbation.

Original: Q2: Does dynamics augmentation improve the performance of the algorithm?
Revised: Q2: Does dynamics augmentation improve the performance of the algorithm?

Original: If so, how much should we perturb the original data?
Revised: If so, how much should we perturb the original data?

Original: We run similar experiments on environment N5-C100 with CD-PPO, in which the local simulator is ingested with a mixture of original dynamics data and LSTM generated data.
Revised: We run similar experiments on environment N5-C100 with CD-PPO, in which the local simulator is ingested with a mixture of original dynamics data and LSTM generated data.

Original: The ratio of perturbed dynamics data varies from 0% to 100%.
Revised: The ratio of perturbed dynamics data varies from 0% to 100%.

Original: And we found that the algorithm turns out the best performance when we use fully generated data, as shown in Figure 8.
Revised: And we found that the algorithm turns out the best performance when we use fully generated data, as shown in Figure 8.