Original: Centralized vs Individual Models for Decision Making in Interconnected Infrastructure
Revised: Centralized vs Individual Models for Decision Making in Interconnected Infrastructure

Original: Anonymous Authors

Original: Abstract
Revised: Abstract

Original: The 2013 National Infrastructure Protection Plan (DHS, 2013) outlines the need for interconnected infrastructure systems to coordinate more and recognize their interdependencies.
Revised: The 2013 National Infrastructure Protection Plan (DHS, 2013) outlines the need for interconnected infrastructure systems to coordinate more and recognize their interdependencies.

Original: We model the two extremes of this coordination spectrum using two different multi-agent models: (a) a model called the “centralized model” in which the agents act as one unit in making decisions and (b) a model called the “individual model” in which the agents act completely separately and have either a pessimistic or optimistic assumption regarding the damages of the other infrastructure systems controlled by the other agents.
Revised: We model the two extremes of this coordination spectrum using two different multi-agent models: (a) a model called the “centralized model” in which the agents act as one unit in making decisions and (b) a model called the “individual model” in which the agents act completely separately and have either a pessimistic or optimistic assumption regarding the damages of the other infrastructure systems controlled by the other agents.

Original: We then use the individual model to establish a point along the coordination spectrum by providing the individual agents with delayed information from the other player(s).
Revised: We then use the individual model to establish a point along the coordination spectrum by providing the individual agents with delayed information from the other player(s).

Original: To test this framework, we use a small but illustrative model from a Yu & Baroud paper in which there is a power and a water network, and we assume that there are operators for both networks that would like to maximize flow in the network (2020).
Revised: To test this framework, we use a small but illustrative model from a Yu & Baroud paper in which there is a power and a water network, and we assume that there are operators for both networks that would like to maximize flow in the network (2020).

Original: Our results comparing partially repaired networks using the two models find that: (i) the centralized model acts as an upper bound on the individual model in terms of our flow metric and (ii) the delayed information individual model leads to less variability in results compared to the other individual model assumptions which points to the value of some coordination in decision making. 
Revised: Our results comparing partially repaired networks using the two models find that: (i) the centralized model acts as an upper bound on the individual model in terms of our flow metric and (ii) the delayed information individual model leads to less variability in results compared to the other individual model assumptions which points to the value of some coordination in decision making. Correspondence to: Stephanie Allen < sallen7@umd.edu > . 2nd AI4Science Workshop at the 39 th International Conference on Machine Learning (ICML), 2022.

Revised: Copyright 2022 by the author(s).

Original: 1. Introduction
Revised: 1. Introduction

Original: Modeling the decisions made by those responsible for the components of interconnected infrastructure systems in response to disasters helps us better understand the best ways to approach coordination between these entities.
Revised: Modeling the decisions made by those responsible for the components of interconnected infrastructure systems in response to disasters helps us better understand the best ways to approach coordination between these entities.

Original: Indeed, the first three tenets in the 2013 National Infrastructure
Revised: Indeed, the first three tenets in the 2013 National Infrastructure

Original: Protection Plan (DHS, 2013) state:
Revised: Protection Plan (DHS, 2013) state:

Original: • “Risk should be identified and managed in a coordinated and comprehensive way across the critical infrastructure community to enable the effective allocation of security and resilience resources.”
Revised: • “Risk should be identified and managed in a coordinated and comprehensive way across the critical infrastructure community to enable the effective allocation of security and resilience resources.”

Original: • “Understanding and addressing risks from cross-sector dependencies and interdependencies is essential to enhancing critical infrastructure security and resilience.”
Revised: • “Understanding and addressing risks from cross-sector dependencies and interdependencies is essential to enhancing critical infrastructure security and resilience.”

Original: • “Gaining knowledge of infrastructure risk and interdependencies requires information sharing across the critical infrastructure community.”
Revised: • “Gaining knowledge of infrastructure risk and interdependencies requires information sharing across the critical infrastructure community.”

Original: The first and third items point to the need for coordination between infrastructure components, and the second item elucidates the existence of interdependencies among infrastructure components.
Revised: The first and third items point to the need for coordination between infrastructure components, and the second item elucidates the existence of interdependencies among infrastructure components.

Original: Our paper explores the points brought up in this report by modeling the two ends of the coordination spectrum, with a model in which infrastructure components are completely centralized in their decision making and a model in which the components make their own decisions with either very pessimistic or very optimistic assumptions regarding the other network.
Revised: Our paper explores the points brought up in this report by modeling the two ends of the coordination spectrum, with a model in which infrastructure components are completely centralized in their decision making and a model in which the components make their own decisions with either very pessimistic or very optimistic assumptions regarding the other network.

Original: We also use the individual model to simulate the situation of delayed information sharing between agents, meaning the individual agents still make decisions separately but have knowledge of the previous state of the other players’ network states.
Revised: We also use the individual model to simulate the situation of delayed information sharing between agents, meaning the individual agents still make decisions separately but have knowledge of the previous state of the other players’ network states.

Revised: Overall, as our experiments demonstrate, the centralized model acts as best case scenario model against which to compare the individual models for performance.

Original: 2. Literature Review
Revised: 2. Literature Review

Original: We include literature in this review that pertains to Markov decision process (MDP) models or reinforcement learning techniques applied to interconnected infrastructure in the event of some kind of disaster.
Revised: We include literature in this review that pertains to Markov decision process (MDP) models or reinforcement learning techniques applied to interconnected infrastructure in the event of some kind of disaster.

Original: First, there are singleplayer Markov decision process models for interconnected infrastructure (Espada Jr, 2014) that use linear programming (Huang et al., 2017) along with decomposition techniques (Huang et al., 2018) as well as approximate dynamic programming techniques (Nozhati, 2021;
Revised: First, there are singleplayer Markov decision process models for interconnected infrastructure (Espada Jr, 2014) that use linear programming (Huang et al., 2017) along with decomposition techniques (Huang et al., 2018) as well as approximate dynamic programming techniques (Nozhati, 2021;

Original: Nozhati et al., 2020) and Monte Carlo techniques (Khouj et al., 2014; 2018).
Revised: Nozhati et al., 2020) and Monte Carlo techniques (Khouj et al., 2014; 2018).

Original: These models then move into reinforcement learning (Lopez et al., 2018; Alutaibi, 2017; Sun & Zhang, 2020) and deep reinforcement learning (Ishigaki et al., 2020; Memarzadeh & Pozzi, 2019) techniques for single-player critical infrastructure applications.
Revised: These models then move into reinforcement learning (Lopez et al., 2018; Alutaibi, 2017; Sun & Zhang, 2020) and deep reinforcement learning (Ishigaki et al., 2020; Memarzadeh & Pozzi, 2019) techniques for single-player critical infrastructure applications.

Original: Our concentration is on cooperative, multi-agent MDP and multi-agent reinforcement learning methods for disaster affected interconnected infrastructure systems, which is an underdeveloped literature.
Revised: Our concentration is on cooperative, multi-agent MDP and multi-agent reinforcement learning methods for disaster affected interconnected infrastructure systems, which is an underdeveloped literature.

Original: Some exceptions are a multi-agent reinforcement learning paper (Megherbi et al., 2013)paper and three deep reinforcement learning papers (Rajulapati et al., 2020a;b; Srikanth et al., 2021) that fit this description.
Revised: Some exceptions are a multi-agent reinforcement learning paper (Megherbi et al., 2013) and three deep reinforcement learning papers (Rajulapati et al., 2020a;b; Srikanth et al., 2021) that fit this description.

Original: There are a few additional papers that focus upon non-cooperative infrastructure applications (Panfili et al., 2018; Ni & Paul, 2019).
Revised: There are a few additional papers that focus upon non-cooperative infrastructure applications (Panfili et al., 2018; Ni & Paul, 2019).

Original: In Megherbi et al.
Revised: In Megherbi et al.

Original: , they focus on experiments with reinforcement learning in which the multiple agents involved either did or did not pass information to each other, in which they found that passing information leads to quicker learning (2013).
Revised: , they focus on experiments with reinforcement learning in which the multiple agents involved either did or did not pass information to each other, in which they found that passing information leads to quicker learning (2013).

Original: Our paper is different from Megherbi et al.
Revised: Our paper is different from Megherbi et al.

Original: because: (i) Megherbi et al.
Revised: because: (i) Megherbi et al.

Original: does not truly deal with interconnected infrastructure networks  and (ii) we focus on centralized versus individualized control, along with a delayed information sharing model, thus presenting more cases for interaction among agents  (2013).
Revised: does not truly deal with interconnected infrastructure networks in that they do not model connections between different infrastructure systems and (ii) we focus on centralized versus individualized control, along with a delayed information sharing model, thus presenting more cases for interaction among agents and providing a best case scenario against which to compare models (2013).

Original: Rajulapati et al.’s
Revised: Rajulapati et al.’s

Original: papers use the multi-agent deep deterministic policy gradient (MADDP) algorithm in which the agents involved are trained using “centralized training” and “decentralized execution” techniques, meaning that the agents are provided with global information during the training phase but are not provided with this information after the training phase (2020a; 2020b).
Revised: papers use the multi-agent deep deterministic policy gradient (MADDP) algorithm in which the agents involved are trained using “centralized training” and “decentralized execution” techniques, meaning that the agents are provided with global information during the training phase but are not provided with this information after the training phase (2020a; 2020b).

Original: Srikanth et al.
Revised: Srikanth et al.

Original: use the MADDP algorithm in the context of Covid-19 interconnected infrastructure presumably with the same training scheme (2021).
Revised: use the MADDP algorithm in the context of Covid-19 interconnected infrastructure presumably with the same training scheme (2021).

Original: Our paper differs from these last three papers because they only consider the case of decentralized control and never consider the case in which complete centralization could make the situation better.
Revised: Our paper differs from these last three papers because they only consider the case of decentralized control and never consider the case in which complete centralization could make the situation better.

Original: We also consider network measures of reward, whereas the reward mapping is not explicitly specified in these three papers.
Revised: We also consider network measures of reward, whereas the reward mapping is not explicitly specified in these three papers.

Original: 2.1. Our Contribution
Revised: 2.1. Our Contribution

Original: Our contribution to the literature is to provide some additional models for multi-agent coordination in interconnected infrastructure.
Revised: Our contribution to the literature is to provide some additional models for multi-agent coordination in interconnected infrastructure.

Original: We also shine a light on the policy objective of having interconnected infrastructure systems coordinate more by showing the two ends of the spectrum of coordination along with a point along the continuum where there is delayed information sharing.
Revised: We also shine a light on the policy objective of having interconnected infrastructure systems coordinate more by showing the two ends of the spectrum of coordination along with a point along the continuum where there is delayed information sharing.

Original: Furthermore, we contribute to the literature by explicitly modeling interconnected infrastructure outcomes using a network measurement of reward.
Revised: Furthermore, we contribute to the literature by explicitly modeling interconnected infrastructure outcomes using a network measurement of reward.

Original: 3. Markov Decision Processes
Revised: 3. Markov Decision Processes

Original: Before moving into our multi-agent models, we provide some background regarding Markov decision processes.
Revised: Before moving into our multi-agent models, we provide some background regarding Markov decision processes.

Original: As defined by Sutton and Barto, a Markov decision process for an agent i over T total time steps can be defined by a few sets and functions as follows (2018):
Revised: As defined by Sutton and Barto, a Markov decision process for an agent i over T total time steps can be defined by a few sets and functions as follows (2018):

Original: • S as the set of states in which the agent can exist.
Revised: • S as the set of states in which the agent can exist.

Original: • A as the set of actions the agent can take.
Revised: • A as the set of actions the agent can take.

Original: A ( s ) can be a function of the state the agent in which the agent is currently.
Revised: A ( s ) can be a function of the state the agent in which the agent is currently.

Original: • p ( s ′ , r | s, a ) =
Revised: • p ( s ′ , r | s, a ) =

Original: Pr { R t +1 = r, S t +1 = s ′
Revised: Pr { R t +1 = r, S t +1 = s ′

Original: | S t = s, A t = a } , as the probability of transitioning to state s ′ and obtaining reward r at time t + 1 given that the agent began at state s and took action a at time t .
Revised: | S t = s, A t = a } , as the probability of transitioning to state s ′ and obtaining reward r at time t + 1 given that the agent began at state s and took action a at time t .

Original: This definition expresses the Markov Property of the next state only depending upon the previous state and action.
Revised: This definition expresses the Markov Property of the next state only depending upon the previous state and action.

Original: • r
Revised: • r

Original: ( s, a )=
Revised: ( s, a )

Original: E [ R t +1 | S t

Original: [Equation]
Revised: [Equation]

Original: p ( s ′ , r | s, a ) , as the reward at state s and action a , requiring us to take the expectation over all rewards.
Revised: ) , as the reward at state s and action a , requiring us to take the expectation over all rewards.

Original: • π ( a | s ) as the policy.
Revised: • π ( a | s ) as the policy.

Original: It represents the probability of the agent choosing action a given that the agent is at state s .
Revised: It represents the probability of the agent choosing action a given that the agent is at state s .

Original: • The state-value function is defined, with discount factor γ ∈ [0 , 1) , as:
Revised: • The state-value function is defined, with discount factor γ ∈ [0 , 1) , as:

Original: [Equation]
Revised: [Equation]

Original: • The state-action-value function is defined as:
Revised: • The state-action-value function is defined as:

Original: [Equation]
Revised: [Equation]

Original: The Bellman Equation is defined as the optimal state-actionvalue function for a pair ( s, a ) in the logical progression as follows (Sutton & Barto, 2018):
Revised: The Bellman Equation is defined as the optimal state-actionvalue function for a pair ( s, a ) in the logical progression as follows (Sutton & Barto, 2018):

Original: [Equation]
Revised: [Equation]

Original: We obtain a recursive equation which defines the Bellman equation.
Revised: We obtain a recursive equation which defines the Bellman equation.

Original: The third step in (3) illustrates the greedy nature of the process of finding an optimal policy (Sutton &
Revised: The third step in (3) illustrates the greedy nature of the process of finding an optimal policy (Sutton &

Original: Barto, 2018) because it demonstrates that, as long as one knows the next steps ahead are optimal, one can choose the action that maximizes reward in this time period.
Revised: Barto, 2018) because it demonstrates that, as long as one knows the next steps ahead are optimal, one can choose the action that maximizes reward in this time period.

Original: Indeed, this is the underpinning of the dynamic programming algorithms which suppose that policies can be improved with this greedy procedure (Sutton & Barto, 2018).
Revised: Indeed, this is the underpinning of the dynamic programming algorithms which suppose that policies can be improved with this greedy procedure (Sutton & Barto, 2018).

Original: For our multi-agent models, our transition function is deterministic so, although we presented the MDP model with a probabilistic transition function for the sake of generality, we will define our multi-agent models with a deterministic transition function that still preserves the Markov Property.
Revised: For our multi-agent models, our transition function is deterministic so, although we presented the MDP model with a probabilistic transition function for the sake of generality, we will define our multi-agent models with a deterministic transition function that still preserves the Markov Property.

Original: 4. Multi-Agent Models
Revised: 4. Multi-Agent Models

Original: The definition of a multi-agent Markov decision process (MMDP) model comes jointly from Boutilier (1996) and
Revised: The definition of a multi-agent Markov decision process (MMDP) model comes jointly from Boutilier (1996) and

Original: Oliehoek & Amato (2016) with Boutilier describing it as thinking of the agents as having “one mind” and Oliehoek & Amato describing it “as a regular MDP with a ‘puppeteer’ agent that selects joint actions”.
Revised: Oliehoek & Amato (2016) with Boutilier describing it as thinking of the agents as having “one mind” and Oliehoek & Amato describing it “as a regular MDP with a ‘puppeteer’ agent that selects joint actions”.

Original: Our centralized model flows directly from these two definitions, with the agents acting “as one mind” (Boutilier, 1996) and representing the idealized situation in which the agents would be able to take advantage of “global information” toward a shared goal (Oliehoek & Amato, 2016).
Revised: Our centralized model flows directly from these two definitions, with the agents acting “as one mind” (Boutilier, 1996) and representing the idealized situation in which the agents would be able to take advantage of “global information” toward a shared goal (Oliehoek & Amato, 2016).

Original: The individual model, on the other hand, treats the agents entirely separately, with each agent making its own decisions and receiving its own rewards given either essentially no information about the other player or delayed information about the other player, which will be explained in the more detailed explanation of the model.
Revised: The individual model, on the other hand, treats the agents entirely separately, with each agent making its own decisions and receiving its own rewards given either essentially no information about the other player or delayed information about the other player, which will be explained in the more detailed explanation of the model.

Original: Therefore, we can call the centralized model a multi-agent Markov decision process, but the individual model needs a new term, which we assign as “multiple MDP model” (MuMDP) because it recognizes that multiple single agent MDP models are being solved for multiple agents.
Revised: Therefore, we can call the centralized model a multi-agent Markov decision process, but the individual model needs a new term, which we assign as “multiple MDP model” (MuMDP) because it recognizes that multiple single agent MDP models are being solved for multiple agents.

Original: These models will be defined specifically for our application of an interconnected water and power system, and we consider the specific disaster of an earthquake for the transition functions.
Revised: These models will be defined specifically for our application of an interconnected water and power system, and we consider the specific disaster of an earthquake for the transition functions.

Original: Centralized Model: We define the centralized model as follows:
Revised: Centralized Model: We define the centralized model as follows:

Original: • State S t : A vector that can be split into two parts: S water t and S power t . 
Revised: • State S t : A vector that can be split into two parts: S water t and S power t . Each part has

Original: Each part has three components, with each part the length of the number of nodes of the infrastructure sub-network (water and power in our case).
Revised: three components, with each part the length of the number of nodes of the infrastructure sub-network (water and power in our case)

Original: The three components are: – the initial damage state; – the number of days that have been spent repairing the network; and – the amount of damage the node currently has.
Revised: The three components are : – the initial damage state; – the number of days that have been spent repairing the network; and – the amount of damage the node currently has.

Original: Importantly, there are 5 initial damage states, taken from the FEMA HAZUS Earthquake Manual (FEMA, 2020), that correspond to undamaged, slight, moderate, extensive, and complete (with corresponding numbers 0, 1, 2, 3, and 4, respectively).
Revised: Importantly, there are 5 initial damage states, taken from the FEMA HAZUS Earthquake Manual (FEMA, 2020), that correspond to undamaged, slight, moderate, extensive, and complete (with corresponding numbers 0, 1, 2, 3, and 4, respectively).

Original: • Action A t =
Revised: • Action A t =

Original: A water t × A power t : Represents the joint pair of actions that the water and power operators take to repair nodes in their respective networks.
Revised: A water t × A power t : Represents the joint pair of actions that the water and power operators take to repair nodes in their respective networks.

Original: We assume each time period is a day, and thus the operators can send resources to repair one node per day.
Revised: We assume each time period is a day, and thus the operators can send resources to repair one node per day.

Original: Depending upon the initial damage, this repair will lead to different amounts of repair of damaged components.
Revised: Depending upon the initial damage, this repair will lead to different amounts of repair of damaged components.

Original: • Transition S t +1 =
Revised: • Transition S t +1 =

Original: T ( S t , A t ) : Represents the transition function from S t , A t to S t
Revised: T ( S t , A t ) : Represents the transition function from S t , A t to S t

Original: +1 .
Revised: +1 .

Original: This function adjusts S t according to the A t by adding 1 more day of work to any component that has been chosen by A t .
Revised: This function adjusts S t according to the A t by adding 1 more day of work to any component that has been chosen by A t .

Original: It also adjusts the damage levels using a combination of (a) the initial damage and (b) the number of days that work has been done on each of the nodes.
Revised: It also adjusts the damage levels using a combination of (a) the initial damage and (b) the number of days that work has been done on each of the nodes.

Original: The FEMA HAZUS Documentation (FEMA, 2020) has restoration functions for the water and power systems according to their initial damage categorization, so our code implements these curves.
Revised: The FEMA HAZUS Documentation (FEMA, 2020) has restoration functions for the water and power systems according to their initial damage categorization, so our code implements these curves.

Original: Thus, the correct restoration curve is chosen for the correct component (supply, demand, transition/transmission), and then the number of days of work is plugged into this restoration curve.
Revised: Thus, the correct restoration curve is chosen for the correct component (supply, demand, transition/transmission), and then the number of days of work is plugged into this restoration curve.

Original: These curves each have their own mean and standard deviation regarding the percentage of functionality the network component has attained for the number of days upon which it has been worked.
Revised: These curves each have their own mean and standard deviation regarding the percentage of functionality the network component has attained for the number of days upon which it has been worked.

Original: We use the cumulative distribution functions.
Revised: We use the cumulative distribution functions.

Original: • Reward R ( S t +1 ) : Represents the reward obtained at time t .
Revised: • Reward R ( S t +1 ) : Represents the reward obtained at time t .

Original: We calculate the reward based upon the maximum amount of flow that can be sent between: – The water supply and water demand nodes (water flow); – The power supply and power demand nodes (power flow); and – The water supply and power demand nodes (water flow) and the power supply and water demand nodes (power flow).
Revised: We calculate the reward based upon the maximum amount of flow that can be sent between: – The water supply and water demand nodes (water flow); – The power supply and power demand nodes (power flow); and – The water demand and power supply nodes (water flow) and the power demand and water supply nodes (power flow).

Original: Importantly, we should note that, based upon the implementation of our algorithms, we do not make the distinction between arcs devoted to water and arcs devoted to power.
Revised: Importantly, we should note that, based upon the implementation of our algorithms, we do not make the distinction between arcs devoted to water and arcs devoted to power.

Original: The arcs in the network are assumed to be able to handle both water and power.
Revised: The arcs in the network are assumed to be able to handle both water and power.

Original: The capacity of the arcs is determined by the nodes incident to them.
Revised: The capacity of the arcs is determined by the nodes incident to them.

Original: The determination occurs by taking the minimum of the repair levels of the two nodes i and j , which produces the capacity level for the arc ( i, j ) .
Revised: The determination occurs by taking the minimum of the repair levels of the two nodes i and j , which produces the capacity level for the arc ( i, j ) .

Original: We use the Python package networkx to determine the maximum flow between the pairs of nodes listed above (Hagberg et al., 2008).
Revised: We use the Python package networkx to determine the maximum flow between the pairs of nodes listed above (Hagberg et al., 2008).

Original: This approach to measuring critical infrastructure repair was inspired by Due˜nasOsorio et al.
Revised: This approach to measuring critical infrastructure repair was inspired by Due˜nasOsorio et al.

Original: Individual Model: The individual model shares elements with the centralized model, but the key difference is that the water and power operators act independently with either a completely pessimistic of each other’s situations, a completely optimistic view of each other’s situations, or delayed information regarding each other’s situations.
Revised: Individual Model: The individual model shares elements with the centralized model, but the key difference is that the water and power operators act independently with either a completely pessimistic of each other’s situations, a completely optimistic view of each other’s situations, or delayed information regarding each other’s situations.

Original: This approach was inspired by Riel et al.
Revised: This approach was inspired by Riel et al.

Original: (2017) and Talebiyan & Duenas-Osorio (2020).
Revised: (2017) and Talebiyan & Duenas-Osorio (2020).

Original: • State S t : There are two separate vectors S water t and S power t .
Revised: • State S t : There are two separate vectors S water t and S power t .

Original: They have the three components discussed in the previous formulation.
Revised: They have the three components discussed in the previous formulation.

Original: • Action A t : There are separate action spaces for water and power instead of a joint action space.
Revised: • Action A t : There are separate action spaces for water and power instead of a joint action space.

Original: Therefore, we have A water t and A power t separately.
Revised: Therefore, we have A water t and A power t separately.

Original: • Transition S water t +1 =
Revised: • Transition S water t +1 =

Original: T ( S water t , A water t ) and S power t + = T ( S power t , A power t ) :
Revised: T ( S water t , A water t ) and S power t + = T ( S power t , A power t ) :

Original: The transition functions perform the same routines as the transition function from the previous set up, but we make it clear that there are separate transition functions for the two players.
Revised: The transition functions perform the same routines as the transition function from the previous set up, but we make it clear that there are separate transition functions for the two players.

Original: • Reward R water ( S water t +1 ) and R power ( S power t +1 ) :
Revised: • Reward R water ( S water t +1 ) and R power ( S power t +1 ) :

Original: This is the most important part of this MuMDP formulation.
Revised: This is the most important part of this MuMDP formulation.

Original: Water and power 
Revised: Water and power have explicitly

Original: have explicitly separate reward functions, in which the water reward function places all 0s, all 1s, or the previous state capacities for the power node capacities and the power reward function places all 0s, all 1s, or the previous state capacities for the water node capacities.
Revised: separate reward functions, in which the water reward function places all 0s, all 1s, or the previous state capacities for the power node capacities and the power reward function places all 0s, all 1s, or the previous state capacities for the water node capacities.

Original: The 0s correspond to each player thinking the other has been completely destroyed by the disaster (pessimistic assumption), the 1s correspond to each player thinking the other sustained no damage after the disaster (optimistic assumption), and the previous state capacities represent delayed information about the other player’s state after the disaster.
Revised: The 0s correspond to each player thinking the other has been completely destroyed by the disaster (pessimistic assumption), the 1s correspond to each player thinking the other sustained no damage after the disaster (optimistic assumption), and the previous state capacities represent delayed information about the other player’s state after the disaster.

Original: These modeling choices were certainly influenced by Talebiyan & Duenas-Osorio’s Judgement Call mechanic which assumes different information assumptions between players (2020).
Revised: These modeling choices were certainly influenced by Talebiyan & Duenas-Osorio’s Judgement Call mechanic which assumes different information assumptions between players (2020).

Original: The reward functions then calculate the maximum amount of flow according to the three tuples outlined in the reward function sub-section of the centralized model for each of the new augmented state vectors.
Revised: The reward functions then calculate the maximum amount of flow according to the three tuples outlined in the reward function sub-section of the centralized model for each of the new augmented state vectors.

Original: 5. Algorithms to Solve Multi-Agent Models
Revised: 5. Algorithms to Solve Multi-Agent Models

Original: As discussed in the beginning of Section 4, we have the centralized MMDP model and the individual MuMDP model.
Revised: As discussed in the beginning of Section 4, we have the centralized MMDP model and the individual MuMDP model.

Original: As discussed by both Boutilier (1996) and Oliehoek & Amato (2016), we can use single agent MDP solution techniques for MMDP models.
Revised: As discussed by both Boutilier (1996) and Oliehoek & Amato (2016), we can use single agent MDP solution techniques for MMDP models.

Original: We can also use single agent
Revised: We can also use single agent

Original: MDP solution techniques for our MuMDP model because this model is comprised of single agent MDPs.
Revised: MDP solution techniques for our MuMDP model because this model is comprised of single agent MDPs.

Original: Consequently, we use policy iteration for the centralized MMDP model and Q learning for the individual MuMDP model (Sutton & Barto, 2018).
Revised: Consequently, we use policy iteration for the centralized MMDP model and Q learning for the individual MuMDP model (Sutton & Barto, 2018).

Original: As described by Sutton and Barto, policy iteration falls in the class of dynamic programming algorithms and, essentially, it works by iteratively updating the value function with the current policy and, then, using the updated value function to find a better policy (2018).
Revised: As described by Sutton and Barto, policy iteration falls in the class of dynamic programming algorithms and, essentially, it works by iteratively updating the value function with the current policy and, then, using the updated value function to find a better policy (2018).

Original: In
Revised: In

Original: Q learning, the algorithm makes use of the Bellman Equation discussed in Section 3 by taking a linear combination of the previous value of the state-action pair and the bracketed part of the last line of equation (3) to produce the following update (Sutton & Barto, 2018):
Revised: Q learning, the algorithm makes use of the Bellman Equation discussed in Section 3 by taking a linear combination of the previous value of the state-action pair and the bracketed part of the last line of equation (3) to produce the following update (Sutton & Barto, 2018):

Original: [Equation]
Revised: [Equation]

Original: See Algorithms 1 and 2 for more details regarding policy iteration applied to the centralized model and Q learning applied to the individual model.
Revised: See Algorithms 1 and 2 for more details regarding policy iteration applied to the centralized model and Q learning applied to the individual model.

Original: We note for the Q learning algorithm (Algorithm 2) the following; when we run the
Revised: We note for the Q learning algorithm (Algorithm 2) the following; when we run the

Original: Q learning algorithm, we run it for a specific S 

Revised: Q learning algorithm, we run it for a specific S

Original: so, for every iteration k , the S 0 is the same.
Revised: so, for every iteration k , the S 0 is the same.

Original: This means that the Q learning algorithm does not have a set of initialized states from which it has to start searching; it repeatedly starts from one initial state (Sutton & Barto, 2018).
Revised: This means that the Q learning algorithm does not have a set of initialized states from which it has to start searching; it repeatedly starts from one initial state (Sutton & Barto, 2018).

Original: Thus, for the experiments described in Section 6, Q learning has to be run for each trial.
Revised: Thus, for the experiments described in Section 6, Q learning has to be run for each trial.

Original: We also note for Algorithm 2 that the “current belief regarding state of other network” refers to either the pessimistic, optimistic, or delayed information assumptions discussed in Section 4.
Revised: We also note for Algorithm 2 that the “current belief regarding state of other network” refers to either the pessimistic, optimistic, or delayed information assumptions discussed in Section 4.

Original: 6. Experimental Set-up
Revised: As a final note, readers may notice that we do not engage with deep reinforcement learning (DRL) algorithms for this work.

Original: For the experimental set-up, we produce 10 random damage states for the water and power networks by drawing a random integer number between 0 to4 for each node for each of the 10 trials. These random numbers represent thedamage categories for each sub-node of the water and power systems according to FEMA in the event of an earthquake (FEMA, 2020). We then use the policy iteration Algorithm 1 to solve the centralized MMDP model for three time periods into the future to produce a partially repaired network for each of the 10 damage states. We then use Q-learning (Algorithm 2) to solve the individual MuMDP model for ton & Barto, 2018)
Revised: This is due to time limitations; our previous experience with DRL informed us

Original: Initialize V ( s ) ∈ R , π ( s ) ∈
Revised: Initialize V ( s ) ∈ R , π ( s ) ∈

Original: A ( s ) =
Revised: A ( s ) =

Original: A water × A power ∀ s , θ = 1 e − 6 , and stable =while stable = 0 do ∆ = 100 ; while ∆ > θ do ∆ =for s ∈ S do v = V ( s ) ; s ′ =
Revised: A water × A power ∀ s , θ = 1 e − 6 , and stable =while stable = 0 do ∆ = 100 ; while ∆ > θ do ∆ =for s ∈ S do v = V ( s ) ; s ′ =

Original: T ( s, π ( s )) ;
Revised: T ( s, π ( s )) ;

Original: V ( s ) =
Revised: V ( s ) =

Original: R ( s ′ )
Revised: R ( s ′ )

Original: + γV ( s ′ ) ; ∆ = max(∆ , | v − V ( s ) | ) end for end while stable = 1 ; for s ∈ S do old joint action =
Revised: + γV ( s ′ ) ; ∆ = max(∆ , | v − V ( s ) | ) end for end while stable = 1 ; for s ∈ S do old joint action =

Original: π ( s ) ; π ( s ) = argmax a [ R ( T ( s, a ))
Revised: π ( s ) ; π ( s ) = argmax a [ R ( T ( s, a ))

Original: + γV ( T ( s, a ))] ;
Revised: + γV ( T ( s, a ))] ;

Original: If old joint action ̸ = π ( s ) , then stable = 0 ; end for end while three time periods into the future  to produce a partially repaired network for each of the 10 damage states.
Revised: If old joint action ̸ = π ( s ) , then stable = 0 ; end for end while that DRL algorithms can take some time to tune. In potential future work on this subject, DRL algorithms would be an interesting avenue to explore.

Original: We run the individual MuMDP model for each of the  10  damage states fora pessimistic (all 0s capacity for the opposing network), an optimistic (all 1s capacity for the opposing network), and  a delayed information (the previous capacity state for  the opposing network) perspectives on the opposing networks.
Revised: For the experimental set-up, we produce 10 random damage states for the water and power networks by drawing a random integer number between 0 to 4 for each node for each of the 10 trials.

Original:  We then compare the maximum flow using the same metric as for  the reward for both partially repaired networks under the two models (and for the pessimistic, the optimistic, and the delayed approaches for  the individual MuMDP model). We run our experiment on a small but illustrative network from Yu and Baroud (2020).
Revised: These random numbers represent thedamage categories for each sub-node of the water and power systems according to FEMA in the event of an earthquake (FEMA, 2020). We then use the policy iteration Algorithm 1 to solve the centralized MMDP model for three time periods into the future to produce a partially repaired network for each of the 10 damage states.

Original: See Figure 1.
Revised: We then use Q-learning (Algorithm 2) to solve the individual MuMDP model for three time periods into the future to produce a partially repaired network for each of the 10 damage states.

Original: It has 8 water nodes (in blue and on the left), 8 power nodes (in red and on the right), and 18 total links in the network (including interconnected links between the two networks).
Revised: We run the individual MuMDP model for each of the 10 damage states for a pessimistic (all 0s capacity for the opposing network), an optimistic (all 1s capacity for the opposing network), and a delayed information (the previous capacity state for the opposing network) perspectives on the opposing networks.

Original: The water and power networks each have supply (S), demand (D), andtransmission/transition (T) nodes (Yu & Baroud, 2020). For Algorithm 1, the θ value is set at 1 e − 6 . For Algorithm 2, the parameters values are:
Revised: We then compare the maximum flow using the same metric as for the reward for both partially repaired networks under the two models (and for the pessimistic, the optimistic, and the delayed approaches for the individual MuMDP model).

Original: [-• K =• ϵ = 0 .
 + e 10 k − (0 . 4)( K ) K , which comes from (Yu et al. ) • α = 0 .• γ = 0 .-]
Revised: [-• K =• ϵ = 0 .
 + e 10 k − (0 . 4)( K ) K , which comes from (Yu et al. ) • α = 0 .• γ = 0 .-]We run our experiment on a small but illustrative network

Original: Algorithm 
Revised: Algorithm 2 Q

Original: MuMDP (Sutton & Barto, 2018)
Revised: (Sutton & Barto, 2018)

Original: Initialize Q w ( s, a ) , ∀ s ∈ S , a ∈
Revised: Initialize Q w ( s, a ) , ∀ s ∈ S , a ∈

Original: A w ( s ) , arbitrarily, and
Revised: A w ( s ) , arbitrarily, and

Original: Q w ( terminal state , · ) = 0 and Q p ( s, a ) , ∀ s ∈ S , a ∈
Revised: Q w ( terminal state , · ) = 0 and Q p ( s, a ) , ∀ s ∈ S , a ∈

Original: A p ( s ) , arbitrarily, and Q p ( terminal state , · ) =for k = 1 , K do
Revised: A p ( s ) , arbitrarily, and Q p ( terminal state , · ) =for k = 1 , K do

Original: Initialize S w and S p with S 0 value; for t = 1 , T do
Revised: Initialize S w and S p with S 0 value; for t = 1 , T do

Original: Choose A w from A w ( S ) using policy derived from
Revised: Choose A w from A w ( S ) using policy derived from

Original: Q w (such as ϵ -greedy);
Revised: Q w (such as ϵ -greedy);

Original: Choose A p from A p ( S ) using policy derived from
Revised: Choose A p from A p ( S ) using policy derived from

Original: Q p (such as ϵ -greedy);
Revised: Q p (such as ϵ -greedy);

Original: Take action A w , observe S w ′ , and obtain R w by concatenating S w ′ with current belief regarding state of other network;
Revised: Take action A w , observe S w ′ , and obtain R w by concatenating S w ′ with current belief regarding state of other network;

Original: Take action A p , observe S p ′ , and obtain R p by concatenating S p ′ with current belief regarding state of other network; 
Revised: Take action A p , observe S p ′ , and obtain R p by concatenating S p ′ with current belief regarding state of other network; [Equation] S

Original: [Equation]
Revised: [Equation]

Revised: ; end for end for from Yu and Baroud (2020).

Original: 7. Results
Revised: See Figure 1. It has 8 water nodes (in blue and on the left), 8 power nodes (in red and on the right), and 18 total links in the network (including interconnected links between the two networks). The water and power networks each have supply (S), demand (D), and transmission/transition (T) nodes (Yu & Baroud, 2020). For Algorithm 1, the θ value is set at 1 e − 6 .

Revised: For Algorithm 2, the parameters values are:

Original: {+• K =• ϵ = 0 .
+}
Revised: {+• K =• ϵ = 0 .
+}

Revised: + e 10 k − (0 . 4)( K ) K , which comes from (Yu et al., 2021)

Original:  The results in Tables 1, 2, and 3 show that the centralized model solution maximum flow acts as an upper bound on the individual model solution maximum flow.
Revised: • α = 0 .• γ = 0 . 7. Results The results in Tables 1, 2, and 3 show that the centralized model solution maximum flow acts as an upper bound on the individual model solution maximum flow.

Original: This makes intuitive sense because, in the centralized model, the agents are acting as one unit whereas, in the individual model, the agents are not, which does not allow them to take advantage of coordination opportunities to maximize flow in the interconnected arcs/nodes.
Revised: This makes intuitive sense because, in the centralized model, the agents are acting as one unit whereas, in the individual model, the agents are not, which does not allow them to take advantage of coordination opportunities to maximize flow in the interconnected arcs/nodes.

Original: Figure 2, demonstrates that, among the individual models, the delayed information model leads to less variability in the percentage difference metric between the individual model and the centralized model.
Revised: Figure 2, demonstrates that, among the individual models, the delayed information model leads to less variability in the percentage difference metric between the individual model and the centralized model when the ranges of the metric are compared.

Original: This indicates that there is value in delayed information sharing between agents, which requires very little coordination and perhaps suggests an important policy idea in improving disaster management practices between interconnected networks.
Revised: This indicates that there is value in delayed information sharing between agents, which requires very little coordination and perhaps suggests an important policy idea in improving disaster management practices between interconnected networks.

Revised: We do note that the fourth quartiles of the pessimistic and optimistic individual models lie above the fourth quartile of the delayed information individual model but, overall, if we have to choose one model and aim to guard against comparatively large negative outliers, the experiments still point to the delayed information model because its first quartile lies well above the two other models’ first quartiles.

Original: As a note on timing for the experiments, these are the average timing results in seconds for the four experiments undertaken:
Revised: As a note on timing for the experiments [Table] these are the average timing results in seconds for the four experiments undertaken:

Original: • Policy Iteration for the MMDP/Centralized Model: 3976 seconds • Q Learning for the MuMDP/Individual Model with 
Revised: • Policy Iteration for the MMDP/Centralized Model: 3976 seconds • Q Learning for the MuMDP/Individual Model with Pessimistic Assumption: 178 seconds • Q Learning for the MuMDP/Individual Model with Optimistic Assumption: 562 seconds • Q Learning for the MuMDP/Individual Model with Delayed Information Assumption: 545 seconds

Original: [Table]
Revised: [Table]

Original: Pessimistic Assumption: 178 seconds • Q Learning for the MuMDP/Individual Model with Optimistic Assumption: 562 seconds • Q Learning for the MuMDP/Individual Model with Delayed Information Assumption: 545 seconds
Revised: [Table]

Original: 8. Conclusions & Future Work
Revised: 8. Conclusions & Future Work

Original: In conclusion, this work demonstrates the two ends of the spectrum regarding coordination in interconnected infrastructure in the aftermath of a disaster as well as a point on this spectrum in the form of delayed information sharing.
Revised: In conclusion [Table] this work demonstrates the two ends of the spectrum regarding coordination in interconnected infrastructure in the aftermath of a disaster as well as a point on this spectrum in the form of delayed information sharing.

Original: We propose two models, the centralized and the individual model, that span this spectrum as well as a modified version [Table]
Revised: We propose two models, the centralized and the individual model, that span this spectrum as well as a modified version of the individual model with delayed information sharing, and we demonstrate all of these models using a 10 trial experiment.

Original: [Table]
Revised: From this experiment, we see that the centralized model acts as an upper bound on the individual model

Original: [Table]
Revised: [Table]

Original: of the individual model with delayed information sharing, and we demonstrate all of these models using a 10 trial experiment. From this experiment, we see that the centralized model acts as an upper bound on the individual model, and we showcase the value in at least limited coordination in the form of delayed information sharing.
Revised: and we showcase the value in at least limited coordination in the form of delayed information sharing.

Original: In the future, we would test the two models on a larger network, which would require more sophisticated MDP and  reinforcement learning solution techniques.
Revised: In the future, we would test the two models on a larger network, which would require more sophisticated MDP and [Table] reinforcement learning solution techniques.

Original: We would also test more realistic earthquake damage disaster situations, as opposed to the randomized damage scenarios we used for our experiments.
Revised: We would also test more realistic earthquake damage disaster situations, as opposed to the randomized damage scenarios we used for our experiments.

Revised: Finally, we would explore deep reinforcement learning algorithms applied to this context.

Revised: 9. Acknowledgements

Revised: We would like to thank Dr. Allison Reilly from University of Maryland, College Park for a very insightful conversation in Spring 2022 that helped inspire this work.

Revised: She pointed us toward the NIPP 2013 (DHS, 2013) and informed us of the lack of communication between different infrastructure entities.