Near-optimal decentralized algorithms for network dynamic optimization
Abstract
We study dynamic decision-making problems in networks under stochastic uncertainty about future payoffs.
The network has a bounded degree, and each node takes a discrete decision at each period, leading to a per-period payoff that is a sum of three parts: node rewards for individual node decisions, temporal interactions between individual node decisions from the current and previous periods, and spatial interactions between decisions from pairs of neighboring nodes.
The objective is to maximize the expected total payoffs over a finite horizon.
We propose a decentralized algorithm whose computational requirement is linear in the graph size and planning horizon, and characterize sufficient conditions under which our decentralized algorithm achieves near optimality compared to the centralized global optimal.
The class of decentralized algorithms is parameterized by locality parameter L .
An L -local algorithm makes its decision at each node v based on current and (simulated) future payoffs only up to L periods ahead, and only in an L -radius neighborhood around v .
Given any permitted error ϵ > 0 , with L = O (log(1 /ϵ )) , we show that L -local algorithm has an average per-node-per-period optimality loss of up to ϵ when temporal and spatial interactions are relatively small compared to the randomness in the node rewards and the graph degree.
1. Introduction
Many real-world contexts call for dynamic decision making in networks with uncertainty about the future:
At each period, a decision is made at each node in the network and a central planner aims to maximize the total payoff across the network.
Examples of such settings include influence maximization in social networks [5, 34], multi-product pricing on product networks [8, 10] and logistics planning on transportation networks [13, 17].
In all these settings, the resulting payoffs arise from both individual decisions at each node and interactions among neighbors on the network.
Moreover, when the decisions are made repeatedly over time, the current decision at each node influences the future payoff at the node.
Besides the goal of maximizing payoffs in such contexts, it is desirable to have decision rules that are “simple” in various ways such as computational efficiency, potential to be computed in a distributed manner, interpretability, and robustness to model misspecification.
In a networked optimization setting, an attractive class of algorithms is decentralized algorithms which obtain the decisions of individual nodes based solely on information from the “nearby” part of the network [31].
Motivated by the ubiquitous setting of dynamic decision-making on networks and the practicality of decentralized algorithms, we aim to answer the following research question:
Can decentralized algorithms be near-optimal in terms of maximizing collective rewards on networks under stochastic uncertainty about the future?
In this paper, we propose a benchmark model of dynamic decision-making on bounded degree graphs with the global payoff consisting of: 1) per-period individual node rewards, which are random functions over individual decisions; 2) per-period spatial interactions between neighboring nodes, which are random functions of pairs of decisions; 3) temporal interactions for each individual node between consecutive periods, which are also random functions of pairs of decisions.
Contribution.
We show that when temporal and spatial interactions are small with respect to the randomness in node rewards and the graph degree, we can construct a simple class of global nearoptimal decentralized algorithms, which has computation requirement linear in the network size and the time horizon.
For each focal node v , our algorithm simulates future reward realizations up to L periods ahead for a L -radius local neighborhood of the focal node, where L is the locality parameter determined based on desired precision.
Our algorithm then solve the network optimization problem on the local graph and treats only the decision at the focal node as final.
To the best of our knowledge, this is the first result on establishing sufficient conditions for approximating global optimal solutions via decentralized algorithms for network dynamic optimization problems under stochastic uncertainty.
Our work is inspired by the literature on correlation decay for static networks [21, 23], which studies how the effects of decisions at the boundary of a graph propagate towards the focal node.
The correlation decay property has only been previously studied in static settings where reward functions consisting of node rewards and pairwise spatial interactions.
However, when generalizing to dynamic setting, the value-to-go functions contain interactions between groups of nodes which are not neighbors of each other.
Due to such interaction-at-a-distance , previous technical machinery used to establish correlation decay in static networks does not generalize to our dynamic setting.
We develop a novel machinery to establish correlation decay in dynamic decision-making settings (outlined in Appendix C), which handles the interaction-at-a-distance phenomenon.
Related work.
Our paper contributes to the following related research areas: (1) Correlation decay for decentralized algorithms; (2) Dynamic optimization in networks and (3) Multi-agent reinforcement learning.
We include a detailed section on related literature in the Appendix B.
Notation and terminology.
We denote our underlying graph as G =
( V, E ) with node set V and edge set E .
For two nodes u, v ∈ G , we let dist G ( u, v ) denote the length of a shortest path between u and v .
If uv ∈ E , we say u is a neighbor of v .
For any node v ∈ V , we denote by Γ( v ) its set of neighbors: Γ( v ) :
= { u ∈ V : uv ∈ E } ; and denote by d G ( v ) :=
| Γ( v ) | its degree in G .
We let d G denote the degree of graph G , which is the maximum degree of nodes in graph G .
For a subgraph
M of G and a vector y :
= { y v } v ∈ V , we denote by y M the subvector { y v } v ∈ V ( M ) .
Let B G ( v, R ) denote the subgraph induced by all vertices whose distance to v is at most R .
Often times, when the underlying graph is clear from context, we drop the subscript for the above notations.
Given a graph G =
( V, E ) and time horizon T , the spatial-temporal (ST) graph is constructed by making a clone of G for each time t = 1 , 2 , · · · , T , and connecting copies of the same node between consecutive times via edges.
The ST graph distance is defined as follows: given two ST nodes ( v 1 , t 1 ) and ( v 2 , t 2 ) , dist st (( v 1 , t 1 ) , ( v 2 , t 2 ))
=
dist ( v 1 , v 2 ) + | t 1 − t 2 | .
For a given integer K ≥ 1 , we use [ K ] as a short-hand notation for set { 1 , 2 , · · · , K } .
For a collection of random variables Y [ k ] , σ ( Y [ k ] ) denotes the smallest sigma algebra generated by Y [ k ] .
Given a random variable X , we write (cid:101) X
d = X to define (cid:101) X as an independent copy of X which follows the same distribution.
2. Model
We consider a dynamic decision network ( G =
( V, E ) , Φ , T , A , x 0 ) with future stochastic uncertainty.
Here, G is the underlying undirected graph where individual decisions are made at each node.
Φ denotes the joint stochastic reward functions over the graph G during the planning horizon.
We consider a discrete-time model from time 0 to the planning horizon T .
We denote by A the discrete action set that the decision of each node must be chosen from.
The initial decision vector taken on the network is given and is denoted by x 0 ∈ A | V | .
The global objective is to maximize the collective payoff from the entire graph over the time horizon.
At time t , the single-period reward is the sum of three types of (random) reward functions:
• Node reward :
Each node v ∈ V earns a random reward Φ vt ( x vt ) :
A → R , which depends on its decision x vt at time t .
• Temporal interaction reward :
Each node v ∈ V at each time period t is associated with a random reward function Φ v t − 1 ,t ( x v t − 1 , x v t ) :
A×A → R , which capture how consecutive decisions at node v interact with each other.
• Spatial interaction reward : Each edge uv ∈ E at each time period t is associated with a random reward function Φ u,vt ( x ut , x vt ) :
A × A → R , which capture how neighboring nodes interact with each other at time t .
Collectively, we let Φ := {{ Φ vt } t ∈ [ T ] ,v ∈ V , { Φ vt − 1 ,t } t ∈ [ T ] ,v ∈ V , { Φ u,vt } uv ∈ E } denote joint random reward functions.
Given any subgraph M , let Φ Mt :
= {{ Φ vt } v ∈ M , { Φ vt − 1 ,t } v ∈ M , { Φ u,vt } uv ∈ M } .
At each time t ∈ [ T ] , node v ∈ V makes a decision x vt ∈ A := { 0 , 1 , · · · , |A|− 1 } .
These reward functions are endowed with a probabilistic structure: their function values are assumed to follow known distributions.
The random functions Φ node t
[Equation]
{ Φ vt
[Equation]
and Φ inter t
[Equation]
are realized only at the beginning of time period t .
We denote the realized reward functions as { ϕ vt } v ∈ V , { ϕ u,vt } uv ∈ E , and { ϕ vt − 1 ,t } v ∈ V .
Moreover, we denote the reward distribution and realization at time t collectively by Φ t and ϕ t , respectively.
We call x t := { x vt } v ∈ V a decision vector at time period t .
At each time period t , a decision vector x t must be chosen after observing ϕ t .
We illustrate the dynamics under our model through an example in Figure 2.
Given realized reward functions ϕ t at time t , and decision vectors x t − 1 , x t , the single-period reward collected at period t is
[Equation]
The overall goal is to construct a dynamic decision-making policy x t , which is adapted to the available information up to time t , i.e., x t ∈ σ ( x 0 , x [ t − 1] , Φ [ t ] ) , that maximizes the expected collected rewards over the time horizon: R := E Φ (cid:2) (cid:80) T t =1 f t ( x t ; x t − 1 , Φ t ) (cid:3) .
Following the modeling convention on online stochastic optimization [6, 9], we assume that Φ t is independent of past reward functions { ϕ [ t − 1] } .
At period t , we observe the previous decision x t − 1 and reward realization at time t , i.e., ϕ t .
By the principle of optimality, the optimal x t ( x t − 1 , ϕ t ) maximizes the realized value-to-go function: RV t − 1 ( x t ; x t − 1 , ϕ t ) : [Equation] f t [Equation]
; x t −
[Equation]
ϕ
[Equation] + [Equation] t ( x t ; ϕ t ) , where the expected value-to-go function V t ( x t ) is recursively defined as
[Equation]
with V T ( x T ) =
at the end of the horizon.
We denote by x ∗ :
= { x ∗ t } 1 ≤ t ≤T the optimal solution of (2) and denote the optimal expected global reward as R ∗ := E Φ (cid:104) (cid:80) T t =1 f t ( x ∗ t ; x ∗ t − 1 , Φ t ) (cid:105) .
For any adaptive algorithm which makes decisions Alg t ∈ σ ( x 0 , x [ t − 1] , Φ [ t ] ) at time t , we define the expected total rewards under Alg as R ( Alg ) :
= E Φ (cid:104) (cid:80) T t =1 f t ( Alg t ; Alg t − 1 , Φ t ) (cid:105) , with Alg 0 = x 0 .
3. Main Results and Algorithms
Assumption 1 For some constants C node , g, c time , c edge ∈ (0 , ∞ ) , the distributions of reward functions { Φ t } t ∈T satisfy the following:
• For every v ∈ V and t ∈ [ T ] , sup a ∈A | Φ vt ( a ) | ≤ C node .
• For every v ∈ V and t ∈ [ T ] , Φ t are independent of past decisions x
[ t − 1] and past reward realizations ϕ [ t − 1] .
• There exists a constant g > 0 such that for any v ∈ V , t ∈ [ T ] , decisions a ̸ = a ′ ∈ A , given Φ inter t and { Φ ut } u ̸ = v ,
[Equation] [Equation]
) , for any b
[Equation]
< b [Equation]
• With probability 1, for any v ∈ V , ( u, v ) ∈ E , t ∈ [ T ] and decisions a ̸ = a ′ ∈ A , the temporal interaction Φ vt − 1 ,t ( a, a ′ ) [-
-]∈
[ − c time , c time ] , and the edge interaction Φ u,vt ( a, a ′ ) ∈ [ − c edge , c edge ] .
Moreover, we require the constants to satisfy
[Equation]
3.1. Main theorem
[-Deﬁnition 
-]Definition 1 An algorithm for the dynamic decision network ( G, Φ , T , A , x 0 ) is said to be an
L -local algorithm if the decision of node v at time t only relies on the local information up to its Lradius neighborhood in the ST graph, i.e., x vt ∈ σ ( x B ( v,L ) t −, Φ B ( v,L ) t , (cid:101) Φ B ( v,L ) t + · · · , (cid:101) Φ B ( v,L ) t + L ) , where each (cid:101) Φ B ( v,L ) t ′ d =
Φ B ( v,L ) t ′ for t + 1 ≤ t ′ ≤ t + L is an independent copy of Φ B ( v,L ) t ′ .
Note that L is a parameter in both spatial and temporal dimension.
In an L -local algorithm, although the future realizations of reward functions are not revealed, L -step simulations are used to approximate the future value-to-go functions in the corresponding local neighborhood.
Definition 2 Consider a dynamic decision network ( G, Φ , T , A , x 0 ) .
An algorithm Alg is an ϵ (- additive)-approximation algorithm if R ∗ − R ( Alg ) ≤ | V |T ϵ, where R ∗ is the optimal payoff, and
R ( Alg ) is the payoff collected by Alg .
Note that there is a | V |T factor in the loss permitted because the total reward scales up linearly with the number of nodes | V | times the time horizon T ; in other words, we permit an average per-node-per-period loss of up to ϵ .
We introduce a model parameter C which is the largest possible change in total rewards when one node changes its decision at one time.
For any a, a ′ ∈ A , changing from x vt = a to x vt = a ′ can cause at most 2 · C node difference in the node reward, d · 2 c edge difference in the edge rewards, and 2 · 2 c time difference in the temporal rewards.
Hence, we define the constant
[Equation]
Theorem 3 Consider a dynamic decision network ( G, Φ , T , A , x 0 ) where underlying graph G has degree
d ≥ 2 .
Suppose the reward functions Φ satisfy Assumption 1.
Then, given any ϵ > 0 and
L ≜ ⌊ log 2 4 Cϵ ⌋ , we can construct an L -local algorithm for the dynamic decision network problem that is an ϵ -approximation algorithm.
The main contribution of Theorem 3 is to establish the global near-optimality property of a decentralized algorithm under Assumption 1.
Our local algorithm (presented in Algorithm 1) has the advantage of being computationally efficient: the computational requirement of Algorithmis O ( | V |T e poly ( 1 ϵ ) ) , where the dependence on model parameters d, g, C and |A| is suppressed in the O ( · ) notation.
The proof of Theorem 3 and the details on the computational requirement are presented in Appendix C and Appendix G, respectively.
3.2. Local Algorithm
In this section, we present our local algorithm.
Given t ∈ [ T ] , the global decision problem is
[Equation]
The algorithm, outlined in Algorithm 1, determines the decision of each node by solving a decentralized version of (4).
For each node v , the local algorithm utilizes all available reward information from its local neighborhood B ( v, L ) and fixes the decision of each boundary node u ∈
B ( v, L ) \ B ( v, L − 1) as the default decision 0.
We define f Lt ( x t ; x t − 1 , ϕ t ) :
=
(cid:80) u ∈ B ( v,L ) ( ϕ ut ( x ut )+ ϕ ut − 1 ,t
( x ut − 1 , x ut ))+ (cid:80)
uu ′ ∈ B ( v,L ) ϕ u,u ′ t ( x ut , x u
[Equation]
t ) as the single-period payoff on B ( v, L ) and V Lt ( x t ) :=
E Φ t +1 [max x t +1 f
Lt +1 ( x t +1 ; x t , ϕ t +1 )
+ V Lt +1 ( x t +1 )] (with terminal condition V L min { t + L, T } = 0 ) as the expected value-to-go function up to L -step look-ahead on B ( v, L ) .
We omit the dependency on v as the focal node for the decentralized algorithm is usually clear from context.
In addition, we denote by (cid:98) V L,nt ( x t ) the sample average estimate of V Lt ( x t ) by simulating independent samples of Φ t +1 , Φ t +2 , · · · , Φ min( T ,t + L ) .
4. Numerical Experiments
To test the performance of our local algorithm and the presence (or absence) of correlation decay while varying interaction strength, we conducted a simulation experiment.
We summarize the simulation environment and main findings here and defer the details to Appendix I.
In our experiment, we first generate multiple dynamic decision network, parameterized by interaction strength c in both the temporal and spatial dimensions.
These decision networks share all other components (e.g., a random 3 -regular graph as the underlying graph, binary action set
A = { 0 , 1 } , uniform distribution on [ − 1 , 1] as the node reward distribution when taking action 1 ) so that the differences in performance can be solely attributed to the interaction strength.
For each decision network, multiple instances are generated by sampling the first period node rewards (allowing us to compute confidence intervals for the performance).
Then, for each instance, we compute
Algorithm Input: decision Output:
1: set the locality parameter L = ⌊ log 2 4 Cϵ ⌋ and sample size n = O (( 1 ϵ
2log 2 d ) 2: for all v ∈ V do 3: restrict to subgraph B ( v, L ) 4: let y t ∈ A B ( v,L ) be an optimal solution to the following problem
[Equation]
where (cid:98) V L,nt ( · ) is an estimate of V t ( · ) defined recursively in function (cid:98) V L,nτ ( · ) for t ≤ τ ≤ t + L 5: set Alg v t
= y vt 6: end for 1: function (cid:98) V L,nτ ( x τ )
▷ Input : L , n , v , τ .
2: if τ = min { t + L, T } then 3: set (cid:98) V L,nτ ( x τ ) = [-
-]0 for any x τ ∈ A B ( v,L ) 4: else 5: sample { ϕ ( s ) τ +1 } s ∈ [ n ] independently from Φ τ + 6: for any x τ ∈ A B ( v,L ) , compute (cid:98) V L,nτ ( x τ ) :
= 1 n (cid:80) ns =1 (cid:99) RV L,n τ ( x τ ; ϕ ( s ) τ +1 ) where the summand (cid:99) RV L,n τ ( x τ ; ϕ ( s ) τ +1 ) is defined as in Equation (5) with t = τ + 1 .
7: end if 8: end function several solutions: one being the solution to the global optimization problem, and the others obtained by our local algorithms with different choices of the locality parameter.
We formulate each global or local network optimization problem as a Mixed Integer Program (MIP) and solve it through Gurobi [22].
Lastly, for each solution, we compute its relative payoff , which is the ratio between the total payoff under the local solution to that under the global optimal solution.
We summarize the results in Figure 1, with one plot showing simply the relative payoffs (the higher, the better) and the other showing the relative payoff gaps ( 1 − relative payoff) in log scale (the lower, the better).
Our experimental results corroborate our theoretical finding in Theorem 3; when the interaction strength is small (or even medium-sized), the error in payoffs is seen to decay exponentially in the locality parameter.
This is especially prominent on the second plot in Figure 1.
In addition, we observe that when the interaction strength is large c ≥ 0 .
4 , the optimality gap ceases to improve (and remains non-trivial) for locality parameter values larger than 4 .
5. Concluding Remarks
We introduced a benchmark model of a dynamic optimization problem in networks where the global payoff includes spatial interactions and temporal interactions as well as node rewards.
At each time step, a decision vector has to be chosen before observing the realizations of future rewards.
We propose a class of (computationally efficient) decentralized algorithms – which make decisions only using information about the nearby part of the network.
We showed that if spatial and temporal interactions are relatively weak, then the decision maker can employ decentralized algorithms, which essentially optimize on small subgraphs of the network, to first estimate the value-to-go and then to obtain near-optimal decisions
Appendix B. Related Literature
Correlation decay for decentralized algorithms.
Correlation decay is the cornerstone for the success of numerous decentralized algorithms for static network optimization in the literature.
It was first studied in the domain of statistical physics.
The seminal work of Dobrushin [15] studied graphical models (e.g., a Markov chain is a graphical model on a line graph) on infinite graphs via correlation decay methods, investigating whether the joint distribution –the Gibbs measure– is uniquely determined by the distribution of each random variable conditional on its neighbors.
Since then, the concept of correlation decay has expanded to applications beyond statistical physics [11, 14, 28, 36], including wireless communication [4, 20, 36], combinatorial optimization [19, 21], marginal inference on graphical models [33], etc.
The typical regime for static decision problems under which the correlation decay property holds is when the underlying graph has a bounded degree (i.e., each node interacts with a constant number of other nodes) [19, 21, 36].
However, in our multi-period network model involving uncertainty about the future, there exists an implicit interaction between any pair of nodes due to the fact that their decisions for future time periods could be correlated, resulting in new challenges.
Dynamic optimization in networks.
Dynamic decision making in networks under uncertainty about the future has been studied in a variety of contexts including network revenue management [32], network diffusion models [1, 24, 27], online matching under stochastic arrivals [2, 3, 12, 30], and choosing lockdown policies in a commuting network [16], to name a few.
In most of these works the ideas used (e.g., shadow prices) are very different from ours.
Some of these previous work [3, 7] make use of local decompositions that rely on their specific settings, or requires convexity in the reward functions [26].
We adopt a general framework, towards developing a foundational understanding regarding the sufficiency of decentralized algorithms for obtaining near optimality in dynamic stochastic optimization problems.
Our paper contributes to this literature by providing an important theoretical foundation for decision-making problems on large networks: that is, even though the network interactions may be complicated and evolve over time, considering only the local neighborhood around the focal node already gives near-optimal performance when the strength of interactions are relatively weak.
Multi-agent reinforcement learning.
In the setting of multi-agent reinforcement learning (MARL), computational issues are central due to large global state and decision spaces (exponential in the number of agents).
A promising approach is to exploit local dependency structures (i.e., agents only interact with neighboring agents in the network).
Lin et al.
[25], Qu et al.
[29] consider a class of MARL problems where the evolution of the state has only local dependencies, and propose a localized policy that converges to an approximately stationary point .
In general, multi-agent reinforcement learning is a hard problem and most results focus on convergence to a stationary point if interactions are not too strong.
We contribute to this literature by showing, in a special case where the state transitions are deterministic (i.e., the current state is the previous decision), global near optimality of a local policy under weak interactions.
Our work may serve as a starting point for developing an understanding of sufficient conditions for achieving global near optimality in network
MDP settings.
Appendix C. Proof Outline of Theorem
We first provide some explanation and justification for Assumption 1. 1)
The first assumption gives us a uniform bound for the change in the global reward when any single node switches its single-period decision.
This assumption ensures that there is no single node whose decision at a certain period has a dominant impact on the global reward.
The second assumption demands independence of the rewards across time.
E.g., considering the following the reward functions where
A = { 0 , 1 } , Φ vt (0) = 0 , Φ vt (1) = 1 + ϵ vt where ϵ vt ∼ N (0 , 1) , Φ vt − 1 ,t (0 , 0) =
Φ vt − 1 ,t (1 , 1) =
c , Φ vt − 1 ,t (0 , 1) =
Φ vt − 1 ,t (1 , 0)
= 0 , Φ u,vt (0 , 0) =
Φ u,vt (1 , 1) =
c , and Φ u,vt (0 , 1) =
Φ u,vt (1 , 0) = 0 , this assumption requires the joint distributions of { ϵ vt : v ∈ V } t are independent across time periods.
The third assumption guarantees sufficient randomness in the single node reward function at each period.
E.g., considering the same reward functions, this assumption does not restrict ( ϵ vt ) v ∈ V to be independent across nodes as long as there exists a g > 0 such that for any node v ∈ V , t ∈ [ T ] and b 1 < b 2 , P ( ϵ vt ∈ [ b 1 , b 2 ) | ϵ ut : u ̸ = v ) ≤ g ( b 2 − b 1 ) .
The last assumption requires that interactions are small compared to the graph degree, which is crucial for the correlation decay property to emerge.
In Appendix H, we explicitly construct (static) decision networks with c edge = Θ(1 /d ) which exhibit long-range correlations and show that local algorithms can perform poorly on such networks.
To establish the ϵ -approximation results in Theorem 3, we show that with high probability,
Algorithm 1 takes optimal decisions in two steps.
The main technical contribution is the first step, where we construct a sequence of local dynamic optimization problems with increasing local radius.
We use the term locality loss to refer to the probability of making a suboptimal decision due to fixing the boundary nodes of the local neighborhood to the default decision 0 .
The second step is to bound the probability, termed the sampling loss , of making a suboptimal decision at the focal node as a result of using approximate (local) value-to-go functions estimated from sample averages.
The second step relies on standard techniques such as Hoeffding’s inequality, and we defer the details to
Appendix E.1.
C.1. Bounding the Locality Loss
In this subsection, we bound the loss that is unavoidable from local decision making, even if one is able to perfectly estimate the local value-to-go functions.
We define a sequence of decentralized policies { π ( H ) } H ≥ L , indexed by the locality parameter H .
Note that a policy defines a mapping from available information so far to decisions.
We use π vt ( H ) to denote the decision of node v at time t under policy π ( H ) , and collectively, we use π t ( H ) to denote the decision vector at time t .
When solving for the decision at a focal node v ∈ V and time t ∈ [ T ] , the policy π ( H ) focuses on the subgraph B ( v, H ) .
It makes nodes outside of B ( v, H ) taking the default decision 0 at any time.
Along the temporal dimension, the policy π ( H ) computes V Ht ( · ) , an estimate of the value-togo function, via an H -step look-ahead with the terminal expected value-to-go V H min( T ,t + H ) ( x ) =for all decision vectors x .
Formally, for a given focal node v at time t , π ( H ) solves the following:
[Equation]
where the H -step look-ahead value-to-go V Ht ( x t ; ϕ t ) in the objective is defined recursively via
[Equation]
for t ≤ τ ≤ min { t + H, T } with terminal condition V H min( t + H, T ) ( x ) = 0 for any x .
Recall that C := 2 C node + 2 dc edge + 4 c time and x ∗ is the optimal decision.
In this subsection, probability is over all reward distributions Φ =
(Φ 1 , · · · , Φ T ) .
We write P as a short hand for P Φ .
Proposition {+
+} Given any ϵ > 0 , with L = ⌊ log 2 4 Cϵ ⌋ , we have for any v ∈ V and t ∈ [ T ] ,
[Equation]
Proposition 4 establishes that the probability of π ( L ) making a suboptimal decision at the focal node v is exponentially small in the locality parameter L .
In the remaining subsection, we outline two important lemmas that prove Proposition 4.
For the following, we consider a fixed t ∈ [ T ] and focal node v ∈ V .
We also consider a fixed value of H and compare the node decisions we obtain under policies π t ( H ) and π t ( H + 1) .
That is, we compare the solutions of the optimization problems in (6) when setting the locality parameter as H and H + 1 .
For t ≤ τ ≤ t + H , we let w τ denote the optimal solution of (6) at time τ when the locality parameter is H ; and similarly, we let z τ denote the optimal solution of (6) at time τ when the locality parameter is H + 1 .
Moreover, we use w t − 1 and z t − 1 to denote the decision vectors π t − 1 ( H ) and π t − 1 ( H + 1) , respectively.
Note that { w τ } t +1 ≤ τ ≤ t + H and { z τ } t +1 ≤ τ ≤ t + H are the optimal “tentative” decisions from time t + 1 to time t
+ H under π t ( H ) and π t ( H + 1) .
That is, at time t , π t ( H ) (resp.
π t ( H + 1) ) only executes w t (resp.
z t ) and discards the other decision vectors { w t ′ : t ′ > t } (resp. {
z t ′ : t ′ > t } ).
Since we restrict to adaptive policies, z τ and w τ are random variables which are measurable with respect to σ ( x 0 , x [ τ − 1] , Φ [ τ ] ) .
For convenience, we extend the definition of z uτ (resp.
w uτ ) to the entire network by setting z uτ = 0 for u ∈ V \ B ( v, H + 1) (resp.
w uτ = 0 for u ∈ V \ B ( v, H ) ), and this does not change our original optimization problem in Equation (6).
Recall that ρ := 4 g ( dc edge + 2 c time ) and Γ( v ) denotes the neighbors of v .
Lemma
For time t ≤ τ ≤ t + H, and u ∈ V ,
[Equation]
We first look at a special case to get some intuitive understanding for the above lemma: if both P ( w uτ − 1 ̸ =[-z u ⌧  1 ) and P ( w u ⌧ +
-]
z
uτ − 1 ) and P ( w uτ +1 ̸ = z uτ +1 ) are equal to zero, then Lemma 5 implies P ( w uτ ̸ = z uτ , w Γ( u ) τ =
z Γ( u ) τ ) = 0 .
This reflects the fact given a ST node ( τ, u ) in the ST graph, if all immediate neighbors (i.e., spatial neighbors Γ( u ) , temporal neighbors ( τ − 1 , u ) and ( τ +1 , u )) take the same decisions under π ( H ) and π ( H + 1) , then by principle of optimality, ST node ( τ, u ) take the same optimal decision under the above two policies.
The lemma constitutes the key component of our analysis where we circumvent the challenge of analyzing dynamics with uncertainty about the future.
Instead of bounding the probability of the focal node taking a suboptimal decision when k -hop neighbors ( 2 ≤ k ≤ H ) in the (static) spatial graph take suboptimal decisions, we bound this probability in the ST graph since the interactions among nodes in the ST graph are easier to track.
In the ST graph, a node makes a suboptimal decision only if a spatial or temporal neighbor is fixed suboptimally.
The rigorous proof of Lemma 5 is quite involved.
It argues that the event ( w uτ ̸ =
z uτ , w Γ( u ) τ =
z Γ( u ) τ ) happens only if the difference of node reward functions, i.e., Φ uτ ( w uτ ) − Φ uτ ( z uτ ) falls in a small interval whose length is proportional to c time ( I { w uτ − 1 [-
-]̸ =
z uτ − 1 } +
E Φ τ +
[ I { w uτ +
 ̸
=
z uτ +1 } ]) .
By the third condition in Assumption 1, the probability of the event ( w uτ ̸ = z uτ , w Γ( u ) τ =
z Γ( u ) τ ) is proportional to gc time
( I { w uτ − 1 ̸ =
z uτ − 1 } + E Φ τ +
[ I { w uτ +
 ̸
=
z uτ +1 } ]) , which further leads to the inequality in Lemma 5.
We present the details in Appendix D.1.
After obtaining Lemma 5, we use induction on the ST graph distance to node ( v, t ) to upper bound the probability of making different node decisions under π t ( H ) and π t ( H + 1) .
We illustrate our proof ideas in Figure 4 and defer the proof of Lemma 6 to Appendix D.2.
Let ξ :
= ( d + 2) ρ .
Lemma 6 For t ≤ τ ≤ t + H and u ∈ B ( v, H ) , we have
[Equation]
Then, Proposition 4 is straightforward from Lemma 6 combined with a union bound argument (details in Appendix D.2).
Appendix D. Proof details in Section C
In this section, we present the proof details which are omitted in the proof outline.
t ρ ξρ ξ 2 {+ρ ξ 
 ρ ξ+} 4 ρ ξ 4 ρ ξ 4 ρ t +ρ ξρ ξ 2 ρ ξ 3 ρ ξ 4 ρ ξ 4 ρ t +ρ ξρ ξ 2 ρ ξ 3 ρ ξ 4 ρ t +ρ ξρ ξ 2 ρ ξ 3 ρ
H + 1 =
H    0 spatial distance to v ( v,t ) ξ
· P ( z uτ ̸ = w uτ | z Γ( u ) τ ̸ = w Γ( u ) τ
[Equation]
+
[Equation]
ρ D.1. Proof of Lemma
When solving for (6), in order to compute the optimal decision vector under policy π ( H ) at time step t , i.e., π t ( H ) , it also needs to compute a “tentative decision rule” from time t + 1 to time t + H , which we denote them together by w τ for t ≤ τ ≤ t + H .
Similarly, we denote the optimal decision vector at time t and “tentative decision rule” under policy π
( H + 1) as z τ for t ≤ τ ≤ t + H .
In addition, we define z t − 1 :=
π t − 1 ( H + 1) , w t − 1 := π t − 1 ( H ) .
Note that z τ and w τ are random variables, which are measurable with respect to σ ( x 0 , x [ τ − 1] , Φ [ τ ] ) .
For any node u with dist ( v, u ) > H + 1 , we have defined z uτ = w uτ = 0 .
Thus, we consider node u such that dist ( v, u ) ≤ H + 1 .
Define the local version of f τ ( · ) function concerning node u ,
[Equation]
Let z − u τ (resp., w − u τ ) denote the actions at all nodes other than u under the vector z τ (resp., w τ ).
Since z uτ is the optimal solution at time τ when restricting to the subgraph B ( v, H + 1) and using
H + 1 -step look-ahead,
[Equation]
After rearranging the terms in the above inequality,
[Equation]
Similarly, by optimality of w uτ ,
[Equation]
Hence, we define the following positive random variable which is effectively the optimality gap between switching from action w uτ to z uτ under policy π ( H ) ,
[Equation]
Let A τ denote the event such that w uτ ̸ =
z uτ and w Γ( u ) τ =
[Equation]
Then,
[Equation]
Then, by (9)
[Equation]
we further have
[Equation]
Next we expand out the expressions for the approximate value-to-go functions in ( ♮ ).
Define the following functions over decision vectors at time τ + 1 :
[Equation]
where we omit their dependency on Φ τ +1 to simplify the notations.
Then, we have
[Equation]
and
[Equation]
We again similarly omit the dependency on Φ τ +1 to simplify the notations and note that z τ +1 is an optimal solution for max x ∈A B ( v,H +1)
g τ ( x ) and w τ +1 is an optimal solution for max x ∈A B ( v,H ) h τ ( x ) .
Hence,
[Equation]
where the last inequality is since when z u {+τ +
+} = w u τ +
[Equation]
the four terms on the RHS cancel out.
Hence,
[Equation]
Finally, we have
[Equation]
,w Γ( u ) τ =
z Γ( u ) τ ) ≤ P (
[Equation]
{+∆ uτ I { A τ } ≤ 4 c time ( I { z uτ − 1 ̸ = w uτ − 1 } + E Φ τ +
+}
{+[ I { w uτ +
+}
̸
= z uτ +1 } ])) =
P ( w uτ − 1 ̸ =
z uτ − 1 ) P (0 ≤ ∆ uτ I { A τ } ≤ 4 c time (1 + P ( w uτ +1 ̸ =
z uτ +1 | w uτ − 1 ̸ =
z uτ − 1 )) | w τ − 1 ̸ =
z τ − 1 )
+ P ( w uτ − 1 =
z uτ − 1 ) P (0 ≤ ∆ uτ I { A τ } ≤ 4 c time P ( w uτ +1 ̸ =
z uτ +1 | w uτ − 1 =
z uτ − 1 ) | w τ − 1 =
z τ − 1 ) ≤ P ( w uτ − 1 ̸ =
z
uτ − 1 ) · g · 4 c time (1 + P ( w uτ +1 ̸ =
z uτ +1 | w uτ − 1 ̸ =
z uτ − 1 ))
+ P ( w uτ − 1 =
z
{+uτ − 1 ) · g · 4 c time ( P ( w uτ +
+}
̸
=
z uτ +1 | w uτ − 1 =
z
uτ − 1 ))
=
4 gc time ( P ( w uτ − 1 ̸ =
z
uτ − 1 ) + P ( w uτ +1 ̸ =
z uτ +1 ))
≤ ( P ( w uτ − 1 ̸ =
z
uτ − 1 ) + P ( w uτ +1 ̸ = z uτ +1 )) ρ where the second last inequality is based on the following observation: conditional on previous decisions, previous reward functions, current interactions and node reward functions at other nodes except u , ∆ uτ I { A τ } ∈ [0 , s ] for any s {+≥ 
+} if and only if Φ vτ ( w uτ ) − Φ vτ ( z uτ ) is within some length s interval.
Moreover, the probability of the above event is upper bounded by g multiplied by s due to the third condition in Assumption 1: for any a ̸ = a ′ ∈ A , b 1 < b 2 ,
[Equation]
D.2. Proof of Lemma
We define a new distance metric which is more suitable for the dynamic optimization problem we are interested in.
Denote node v ∈ V at time t ∈ [ T ] as the pair ( v, t ) , which we henceforth call a
ST node .
Define the ST distance between two ST nodes ( v 1 , t 1 ) and ( v 2 , t 2 ) as
[Equation]
In particular, if t 1 = t 2 , then dist st (( v 1 , t 1 ) , ( v 2 , t 2 ))
=
dist ( v 1 , v 2 ) .
We also define another parameter:
[Equation]
Note that ξ ≤ 12 under our assumption that ρ := 4 g ( dc edge + 2 c time ) ≤ 1 2( d +2) .
Recall that Γ( u ) denotes the set of neighbors of u in G .
Before proving Lemma 6, we present the following claim.
Claim 1.
Under the same setting as in Lemma 6, for t ≤ τ ≤ t + H and u ∈ B ( v, H + 1) , we have
[Equation]
Proof of Claim 1.
Let E τ denote the event w uτ ̸ = z uτ given w Γ( u ) τ , z Γ( u ) τ .
Then, we define the following positive random variable as in the proof of Lemma 5 which is effectively the optimality gap between switching from action w uτ to z uτ under policy π ( H ) ,
[Equation]
where the last inequality is because changing node action at u at time τ affects at most d spatial edges and 2 temporal edges.
Since ∆ uτ I { E τ } ≥ 0 , we have the following bound under E τ ,
[Equation]
Moreover, since z uτ is optimal under π ( H + 1) ,
[Equation]
which leads to
[Equation]
Combining these two bounds above, we have
[Equation]
□
Proof of Lemma 6.
We prove the lemma by induction on the ST distance.
By Claim 1 above, for 0 ≤ τ < H and u ∈ B ( v, H + 1) ,
[Equation]
Φ [Equation] This serves as the base case for proof of Lemma 6: when ( u, τ ) satisfies dist st (( v, t ) , ( u, τ )) ≥
H + 1 , Lemma 6 holds.
Suppose that for all k ′ > k for some 0 ≤ k ≤ H , we have that if a node ( u, τ ) satisfies dist st (( v, t ) , ( u, τ )) ≤ k ′ , then P Φ ( w uτ ̸
= z uτ ) ≤ ξ H +1 − k ′ ρ .
For the inductive step, we consider nodes ( u, τ ) with dist st (( v, t ) , ( u, τ )) ≤ k for 0 ≤ k ≤ H .
For the following, to simply the notations, we write P Φ as P .
[Equation]
where the first inequality is by induction hypothesis since the spatial neighbors of u has ST distance to ( v, t ) at most k
+ 1 as well as Claim 1; the second inequality is by Lemma 5; the third inequality is again by the induction hypothesis.
Hence we complete the induction step
[Equation]
□
Proof [Proof of Proposition 4]
We first use Lemma 6 for ( u, τ ) =
( v, t ) and obtain
[Equation]
Then, observe that when the locality parameter H = + ∞ , we obtain the optimal node decision ( x vt ) ∗ .
Then we use a union bound over all locality parameters H which is greater than or equal to
L .
[Equation]
since ξ ≤ 1 / 2 and L = ⌊ log 2 4 Cϵ ⌋ .
Appendix E. Proof details for sample approximation E.1. Bounding the Sampling Loss
In this section, we aim to bound the loss in rewards due to approximating the expected value-to-go function using simulation.
The main result of the subsection is given in Proposition 7, which states that Algorithm 1 obtains the same solution as the local policy π ( L ) with high probability.
Proposition 7 Under the conditions in the Theorem 3, given any ϵ > 0 , there exists a function
N = N ( ϵ, d, g, C ) =
O (( 4 Cϵ ) 2log 2 d g 2 C 4 ) < ∞ such that if sample size n ≥ N , then for any v ∈ V , t ≥ 0 ,
[Equation]
Proof [Proof of Proposition 7] Suppose Alg vt ̸ = π vt ( L ) .
By optimality, ϕ vt ( π vt ( L )
[Equation]
ϕ vt ( Alg vt )
+
[Equation]
t
[Equation]
: x vt
[Equation]
vt ( L
[Equation]
[Equation]
vt (cid:0) f Lt ( x t ) + V Lt ( x t ) (cid:1) ≥ 0 . Similarly,
[Equation]
Therefore
[Equation]
By the third condition in the Assumption [Equation] we have that
[Equation]
By definitions of ∆ − v t and ∆ − v,n above, we have ∆ − v,n + ∆ − v t
[Equation]
Therefore, it suffices to show that
[Equation]
Therefore, it suffices to show that
E
[Equation]
By definition, given any x t + L ∈ A B ( v,L ) , (cid:98) V L,nt + L ( x t + L ) =
V Lt + L ( x t + L ) = 0 .
Hence,
E [
[Equation]
x t + L ( (cid:98) V L,nt +
[Equation]
x t + L
[Equation]
− V Lt +
[Equation]
( x t + L ))]
= 0 .
Next, for t ≤ τ ≤ t + L − 1 , we derive a recursive relation between E [max x τ ( (cid:98) V L,n τ ( x τ ) −
V Lτ ( x τ ))] and E [max x τ +1 ( (cid:98) V
{+Lτ +
+}
( x τ +1 )
{+− V Lτ +
+}
( x τ +1 ))] .
Given any x τ ∈
A B ( v,L ) (cid:98) V L,nτ ( x τ ) − V Lτ
[Table]
We now bound the expectation of later two terms on the right-hand side of the above inequality.
Let Y ( s ) :=
{+max x τ +1 ( f Lτ +
+}
( x τ +1 ; x τ , ϕ ( s ) τ +1 )
{++ V Lτ +
+}
( x τ +1 )) .
Then, { Y ( s ) } 1 ≤ s ≤ n are independent random variables with expectation µ :=
{+E [max x τ +
+}
{+( f Lτ +
+}
( x τ +1 ; x τ , {+Φ τ +1 ) + V Lτ +
 ( x τ +1 ))] . Note that+}
[Equation]
Let (cid:101) m denote the number of nodes in B ( v, L ) . (cid:101) m = 1 + d + · · · + d L ≤ d L +1 d − 1 . Since (cid:101) md/ 2 is the maximum number of edges in B ( v, L ) by the handshaking lemma, we have that
[Equation]
where this lower bound is achieved when all nodes receives the worst possible individual reward − C node , the temporal interactions are − c time and the edge interactions are − c edge .
Similarly we obtain the upper bound, [Equation] Since lb ≤ Y ( s ) ≤ ub, [Equation]
where the first equality is by the property of expectation of non-negative random variables, the first inequality is by Hoeffding’s inequality, the second equality is by change of variables, and the last equality is since
[Equation] Taking expectation of max x τ ( (cid:98) V L,nτ ( x τ ) − V Lτ ( x τ )) , we have
[Equation] where the last inequality is due to C := 2 C node + 2 dc edge + 4 c time ≥ 2 C node + dc edge + 2 c time .
Applying Equation (12) L times for t ≤ τ ≤ t + L − 1 , we have
[Equation]
for n ≥ N ( ϵ, d, g, C ) = 8 πg 2 C 4 L 4 (cid:101) m 2 ϵ. Under the condition in the Theorem 3, let L = ⌊ log 2 4 Cϵ ⌋ . Then,
[Equation]
where O ( · ) notation omits logarithmic factors.
Similarly, for such n ≥ N ( ϵ, d, g, C ) , we have
[Equation]
All together, we have
[Equation]
as desired.
Appendix F. Bounding the total loss
We now show Algorithm 1 achieves a near-optimal total reward by establishing Theorem 3.
Proof [Proof of Theorem 3] As a result of Proposition 4 and Proposition 7, we have that for all v ∈ V , and t ∈ [ T ] ,
[Equation]
Recall the definition of C in Equation (3).
Since the largest possible change in total rewards when switching from one node action to another is upper bounded by
[Equation]
Appendix G. Computation Efficiency
Next we look at the computation requirement of Algorithm 1.
In terms of sampling, Algorithmneeds to simulate n = O (( 1 ϵ ) 2log 2 d ) samples from each of the reward functions { Φ t +1 , Φ t +2 , · · · , Φ t + L } .
We illustrate these sample paths in Figure 5.
The following proposition shows the computation requirement of Algorithm 1 for deciding the action of node v at each time step t .
Proposition 8 The computational requirement of Algorithm 1 is O ( | V |T e poly ( 1 ϵ ) ) , where the model parameters d, g, C and |A| are constants in the O ( · ) notation.
To establish Proposition 8, we prove the following lemma on computation requirement for any L -local ( L ≥ 1 ) algorithm.
Then, the computation needed in Proposition 8 is by letting L =
⌊ log 2 4 Cϵ ⌋ and n = N ( ϵ, d, g, C ) defined in Equation (13).
We defer its proof after introducing the following Lemma.
Lemma [-
-]9 The computation requirement for Alg vt for v ∈ V and t ∈ [ T ] under Algorithm 1 is
LK 2 n where n is the sample size and K =
|A| d L is an upper bound on number of decision vectors to enumerate over for the optimization problem in (5) .
Proof [Proof of Lemma 9] We show this by induction.
Let a τ denote the amount of computation needed to compute (cid:98) V L,nτ ( · ) .
Since (cid:98) V L,nt + L ( · ) = 0 , a t + L = 0 .
Suppose now we obtain (cid:98) V L,nτ ( · ) function with computational effort a τ .
Given a decision vector x τ − 1 and a realization ϕ τ , the optimal x τ can be solved by enumerating all possible decision vectors, whose cardinality is at most K .
Under the assumption that { Φ t } t are independent, we can use the same estimation for (cid:98) V L,nτ ( · ) for different realizations of Φ τ − 1 .
This implies:
[Equation]
where the first K is the number of possible decision vectors x τ − 1 , n is the number of samples, and the second K is the computation needed for enumeration.
Hence, we have a t =
LK 2 n =
L |A| 2 d L n .
Proof [Proof of Proposition 8] Let L = ⌊ log 2 4 Cϵ ⌋ .
Then, we have
[Equation]
Moreover, we let the sample size n = N ( ϵ, d, g, C ) =
O (( 4 Cϵ ) 2log 2 d g 2 C 4 ) defined in Equation (13).
With d, g, C and |A| as constants, by Lemma 9, the computation needed for Algorithmis upper bounded by
[Equation]
Appendix H. Interactions must be small to have correlation decay
In this section, we construct a sequence of static (i.e., single period) decision networks indexed by graph degree d with c edge = Θ(1 /d ) such that there is no near-optimal local algorithm for these networks.
The decision networks we construct satisfy all parts of Assumption 1 except the smallinteraction requirement 4 g ( dc edge + 2 c time ) ≤ 1 2( d +2) .
Thus, our construction justifies the need for the upper bound on the strength of the interactions in Assumption 1.
Admittedly, there is some gap between our assumption c edge ≤ Θ(1 /d 2 ) , and the scale c edge = Θ(1 /d ) at which we show here that long-range correlations arise.
In comparison, previous work [21] also assumed c edge ≤ Θ(1 /d 2 ) to obtain correlation decay in a static random decision network.
In our dynamic setting, we have the same scaling to ensure no long-range correlation.
Definition
A d -regular graph G =
( V, E ) is an γ -edge expander for γ ∈ (0 , 1) if for any S ⊆ V such that | S | ≤ | V | / 2 , the number of edges between S and V \ S (the “cut size”) is at least | S | dγ , i.e., cut ( [Equation] [Equation] [Equation]
dγ .
ϕ t ϕ (1) t + ϕ (2) t + ... ϕ ( n ) t + ϕ (1) t +2 . ϕ ( n ) t + ... ϕ (1) t +2 . ϕ ( n ) t + · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ϕ (1) t + L ... ϕ ( n ) t + L
Construction.
Fix degree d ≥ 35 .
It is well known that there exists m 1 < ∞ , such that for any even integer m with m > m 1 , there is a d -regular graph with m nodes that is a 13 -edge expander [18, 35].
In fact, a random d -regular graph has this property asymptotically almost surely (a.a.s.).
That is, let G be uniformly drawn from random d -regular graphs with m nodes where m ≥ m 1 , then G is a 13 -edge expander almost surely.
We define a static random decision network ( G, Φ) with action set { 0 , 1 } as follows:
• Node rewards: The node rewards Φ v (1) are i.i.d.
from Uniform [ − 1 , 1] ; and Φ v (0) = 0 .
• Edge rewards: The edge rewards are “ferromagnetic”:
[Equation]
where c edge := 6 /d .
Since the constructed decision network is static, there are no temporal interactions in our construction and hence we omit x 0 .
The following claim shows that there does not exist near-optimal local algorithms when the small interaction condition in Assumption 1 does not hold.
Claim 2.
For the decision network ( G, Φ) uniformly drawn from d − regular random graph with m nodes, the optimal action vector is either all 1 s or all
s a.s.
Each of these possibilities arises with probability 1 / 2 .
In particular, the optimal solution has long-range correlations.
In particular, any
L -local algorithm which treats the possible node actions 1 and 0 symmetrically achieves expected payoff at least m/ 3 below the optimal.
Proof [Proof of Claim 2.] Consider any action vector x such that the majority of actions is 1 .
We show that the payoff of x is less than the payoff of all 1 s. Let S be the set of nodes where x takes action 0 .
Since G is a 13 -edge expander a.s., cut ( S ) ≥ | S | d/ 3 .
It follows that the total edge rewards under x is at least c edge | S | d/ 3 = 2 | S | smaller than that under all 1 s. On the other hand the difference between the total node rewards under x and that under all 1 s is (cid:80) v ∈ S
Φ v (1) ≥ −| S | since Φ v (1) [-
-]∈ [ − 1 , 1] , i.e., the total rewards under x is at least 2 | S | − | S | = | S | smaller than the total rewards under all 1 s. Similarly, one can show that for any action vector x such that the majority of actions is 0 , the total reward under x is at least m − | S | smaller than the total reward under all 0 s.
It follows that the optimal solution is either all 1 s or 0 s. Moreover, the optimal solution is all 1 s if (cid:80) v ∈ V ϕ v (1) ≥ 0 and all 0 s otherwise.
Since the distribution of i.i.d Uniform distributiion [ − 1 , 1] is symmetric, each of these above possibilities arises with probability 12 .
Now consider any given L and any L -local algorithm which treats 1 and 0 symmetrically.
By symmetry, each node decision is a priori equally likely to be 1 or 0 .
By symmetry, each node decision is a priori equally likely to be 1 or 0 .
It follows that in a large network, about half the decisions will be 1 and the other half will be 0 under the L -local algorithm.
Formally speaking, the expected number of 1 s is m/ 2 , and the variance in the total number of 1 s is Var [ (cid:80) v ∈ V
I { x v = 1 } ] = (cid:80) v ∈ V Var [ I { x v = 1 } ] = (cid:80) v ∈ V ( 12 ) 2 = 14 m.
Hence for any m > 250 , we know by Chebyshev’s inequality that with probability at least 0 .
9 , the number of 1 s will be in the range | S | ∈ (0 .
4 m, 0 .
6 m ) , i.e., the payoff will be at least 0 .
4 m below the optimal (see the previous paragraph)
Combining, the local algorithm suffers expected payoff loss at least 0 .
9 × 0 .
4 m ≥ m/ 3 .
Appendix I. Missing Details for Section
In the following, we explain in details the simulation environment of our experiment.
There are 5 dynamic decision networks parameterized by interaction strength c for both the spatial and temporal dimensions, with c = 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , and 0 .
These decision networks share all other components, which we list below.
• Graph G : Using the NetworkX package in Python, we randomly generate 2 a 3 -regular graph with 500 vertices.
• Time horizon T : To simplify the simulation and reduce the computational effort, we set the time horizon to be 2 for all decision networks.
• Interaction function : Both the spatial interaction and the temporal interaction are ferromagnetic , meaning that agreeing actions incur a bonus of c , whereas disagreeing actions result in no reward.
• Action set A : We assume a binary action set – that is, for all v ∈ V and t ∈ [ T ] , x vt ∈ { 0 , 1 } , where action zero is viewed as the default action, meaning that Φ vt (0) = 0 .
• Node reward : The random node rewards, for both time periods, when taking action 1 are assumed to i.i
d. and follow the uniform distribution on [ − 1 , 1] .
We sample n 1 = 10 instances for each decision network, where these instances differ in terms of the realized node rewards at the first time period.
Having multiple realizations allow us to compute the confidence intervals for the performance of our algorithm.
We denote by { ϕ v, ( i )} v ∈ V the realized first-period node rewards (when taking action 1 ) in the i -th instance, where each ϕ v, ( i )is sampled according to the node reward distribution, i.e., uniformly from [ − 1 , 1] .
Note that the realized node rewards { ϕ v, ( i )} v ∈ V for each i ∈ [ n 1 ] are shared by all the i -th instances of all decision networks.
To remove the loss in rewards due to sampling, we control the variability in the second period node rewards.
That is, we pre-generate two independent sets of samples of node rewards for the second time period.
The first set contains n 2 ,est = 100 samples, which are used to compute solutions at the first time period; and the second set contains n 2 ,eval = 30 samples, which are used to estimate the total payoff under the solutions computed using the first set of samples.
We denote the node rewards when taking action 1 by { ϕ v, ( i ) 2 ,est } v ∈ V for the i -th sample in the first set and by { ϕ v, ( i ) 2 ,eval } v ∈ V for the i -th sample in the second set.
For each instance, we compute several solutions, with one obtained by solving the global optimization problem, and the others obtained by our local algorithms with different locality parameters.
To solve the network optimization problem, either globally or locally, we write a Mixed Integer
Program (MIP) and solve it through Gurobi [22].
The decision variables of the MIP are:
• node actions for the first time period: { x v 1 } v ∈ V ; • disagreement indicator of neighboring nodes for t = 1 : { y e 1 } e ∈ E ; • node actions for the second time period for each sample j : { x v, ( j )} v ∈ V,j ∈ [ n 2 ,est ] ; • disagreement indicator of neighboring nodes for t = 2 for each sample j : { y e, ( j ) } e ∈ E,j ∈ n
est ] ; • temporal disagreement indicator for each node for each sample j : { y v, ( j ) } v ∈ V,j ∈ [ n 2 ,est ] . And the formulation of our MIP is given below.
[Equation]
s.t.
[Equation]
[Equation]
v
[Equation]
[Equation]
n
[Equation]
[Equation]
[Equation]
v
[Equation]
[Equation]
n
[Equation]
[Equation]
( j )
[Equation]
( j )
[Equation]
y v
[Equation]
( j ) [Equation] [Equation] [Equation]
∀ v ∈ V
[Equation]
e ∈ E
[Equation] j ∈ [ n [Equation] Note that V and E are either nodes and edges of the entire graph when solving for the global optimal solution, or nodes and edges of a local graph when solving for the solution using our local algorithm.
Although the MIP is given for obtaining a first time period solution, a similar MIP can be used to estimate the payoff of a given first time period solution, where we take { x v 1 } v ∈ V as given and replace rewards { ϕ v, ( j ) 2 ,est } with { ϕ v, ( j ) 2 ,eval } .