N EURAL S TOCHASTIC D UAL D YNAMIC P ROGRAMMING
Anonymous authors
Paper under double-blind review
A BSTRACT
Stochastic dual dynamic programming (SDDP) is a state-of-the-art method for solving multi-stage stochastic optimization, widely used for modeling real-world process optimization tasks.
Unfortunately, SDDP has a worst-case complexity that scales exponentially in the number of decision variables, which severely limits applicability to only low dimensional problems.
To overcome this limitation, we extend SDDP by introducing a trainable neural model that learns to map problem instances to a piece-wise linear value function within intrinsic low-dimension space , which is architected speciﬁcally to interact with a base SDDP solver, so that can accelerate optimization performance on new instances.
The proposed Neural
Stochastic Dual Dynamic Programming ( ν -SDDP)
continually self-improves by solving successive problems.
An empirical investigation demonstrates that ν -SDDP can signiﬁcantly reduce problem solving cost without sacriﬁcing solution quality over competitors such as SDDP and reinforcement learning algorithms, across a range of synthetic and real-world process optimization problems.
I NTRODUCTION
Multi-stage stochastic optimization (MSSO) considers the problem of optimizing a sequence of decisions over a ﬁnite number of stages in the presence of stochastic observations, minimizing an expected cost while ensuring stage-wise action constraints are satisﬁed (Birge & Louveaux, 2011; Shapiro et al., 2014).
Such a problem formulation captures a diversity of real-world process optimization problems, such as asset allocation (Dantzig & Infanger, 1993), inventory control (Shapiro et al., 2014; Nambiar et al., 2021), energy planning (Pereira & Pinto, 1991), and bio-chemical process control (Bao et al., 2019), to name a few.
Despite the importance and ubiquity of the problem, it has proved challenging to develop algorithms that can cope with high-dimensional action spaces and long-horizon problems (Shapiro & Nemirovski, 2005; Shapiro, 2006).
There have been a number of attempts to design scalable algorithms for MSSO, which generally attempt to exploit scenarios-wise or stage-wise decompositions.
An example of a scenario-wise approach is Rockafellar & Wets (1991), which proposed a progressive hedging algorithm that decomposes the sample averaged approximation of the problem into individual scenarios and applies an augmented Lagrangian method to achieve consistency in a ﬁnal solution.
Unfortunately, the number of subproblems and variables grows exponentially in the number of stages, known as the “curse-ofhorizon” .
A similar proposal in Lan & Zhou (2020) considers a dynamic stochastic approximation, a variant of stochastic gradient descent, but the computational cost also grows exponentially in the number of stages.
Alternatively, stochastic dual dynamic programming (SDDP) (Birge, 1985; Pereira & Pinto, 1991), considers a stage-wise decomposition that breaks the curse of horizon (F¨ullner &
Rebennack, 2021) and leads to an algorithm that is often considered state-of-the-art.
The method essentially applies an approximate cutting plane method that successively builds a piecewise linear convex lower bound on the optimal cost-to-go function.
Unfortunately, SDDP can require an exponential number of iterations with respect to the number of decision variables (Lan, 2020), known as the “curse-of-dimension” (Bal´azs et al., 2015).
Beyond the scaling challenges, current approaches share a common shortcoming that they treat each optimization problem independently .
It is actually quite common to solve a family of problems that share structure, which intuitively should allow the overall computational cost to be reduced (Khalil et al., 2017; Chen et al., 2019).
However, current methods, after solving each problem instance via intensive computation, discard all intermediate results, and tackle all new problems from scratch.
Such methods are destined to behave as a perpetual novice that never shows any improvement with problem solving experience.
In this paper, we present a meta-learning approach, Neural Stochastic Dual Dynamic Programming ( νSDDP) , that, with problem solving experience, learns to signiﬁcantly improve the efﬁciency of SDDP in high-dimensional and long-horizon problems.
In particular, ν -SDDP exploits a specially designed neural network architecture that produces outputs interacting directly and conveniently with a base
SDDP solver.
The idea is to learn an operator that take information about the speciﬁc problem instance, and map it to a piece-wise linear function in the intrinsic low-dimension space for accurate value function approximation, so that can be plugged into a SDDP solver.
The mapping is trainable, leading to an overall algorithm that self-improves as it solves more problem instances.
There are three primary beneﬁts of the proposed approach with carefully designed components:
i) By adaptively generating a low-dimension projection for each problem instance, ν -SDDP reduces the curse-of-dimension effect for SDDP.
ii)
By producing a reasonable value function initialization given a description of the problem instance, ν -SDDP is able to amortize its solution costs, and gain a signiﬁcant advantage over the initialization in standard SDDP on the two benchmarks studied in the paper.
iii)
By restricting value function approximations to a piece-wise afﬁne form , ν -SDDP can be seamlessly incorporated into a base SDDP solver for further reﬁning solution, which allows solution time to be reduced.
Figure 1 provides an illustration of the overall ν -SDDP method developed in this paper.
The remainder of the paper is organized as follows.
First, we provide the necessary background on
MSSO and SDDP in Section 2.
Motivated by the difﬁculty of SDDP and shortcomings of existing learning-based approaches, we then propose ν
-SDDP
in Section 3, with the design of the neural component and the learning algorithm described in Section 3.1 and Section 3.2 respectively.
We compare the proposed approach with existing algorithms that also exploit supervised learning (SL) and reinforcement learning (RL) for MSSO problems in Section 4.
Finally, in Section 5 we conduct an empirical comparison on synthetic and real-world problems and ﬁnd that ν -SDDP is able to effectively exploit successful problem solving experiences to greatly accelerate the planning process while maintaining the quality of the solutions found.
P RELIMINARIES
We begin by formalizing the multi-stage stochastic optimization problem (MSSO), and introducing the stochastic dual dynamic programming (SDDP) strategy we exploit in the subsequent algorithmic development.
We emphasize the connection and differences between MSSO and a Markov decision process (MDP), which shows the difﬁculties in applying the advanced RL methods for MSSO.
Multi-Stage Stochastic Optimization (MSSO).
Consider a multi-stage decision making problem with stages t = 1 , . .
, T , where an observation ξ t ∼ P t ( · ) is drawn at each stage from a known observation distribution P t .
A full observation history { ξ t } Tt =1 forms a scenario , where the observations are assumed independent between stages.
At stage t , an action is speciﬁed by a vector x t .
The goal is to choose a sequence of actions { x t } Tt =1 to minimize the overall expected sum of linear costs (cid:80)
Tt =1 c t ( ξ t ) (cid:62) x t under a known cost function c t .
This is particularly challenging with feasible constraints on the action set.
Particularly, the feasible action set χ t at stage t is given by
[Equation]
where A t , B t and b t are known functions.
Notably, the feasible set χ t ( x t − 1 , ξ t ) at stage t depends on the previous action x t − 1 and the current stochastic observation ξ t .
The MSSO problem can then be expressed as
[Equation]
where ξ 1 and the problem context U = { u t } Tt =1 for u t :=
( P t , c t , A t , B t , b t ) are provided.
Given the context U given, the MSSO is speciﬁed.
Since the elements in U are probability or functions, the deﬁnition of U is only conceptual.
In practice, we implement U with its sufﬁcient representations.
We will demonstrate the instantiation of U in our experiment section.
MSSO is often used to formulate real-world inventory control and portfolio management problems; we provide formulations for these speciﬁc problems in in Appendix B.1 and Appendix B.2.
Similar to MDPs, value functions provide a useful concept for capturing the structure of the optimal solution in terms of a temporal recurrence.
Following the convention in the MSSO literature (F¨ullner & Rebennack, 2021), let
[Equation]
which expresses a Bellman optimality condition over the feasible action set.
Using this deﬁnition, the
MSSO problem (3) can then be rewritten as
[Equation]
Theorem 1 (informal, Theorem 1.1 and Corollary 1.2 in F¨ullner & Rebennack (2021))
When the optimization (5) almost surely has a feasible solution for every realized scenario, the value functions Q t ( · , ξ ) and V t ( · ) are piecewise linear and convex in x t for all t
= 1 , . .
, T .
Algorithm
[Table]
(cid:8) V 0 t (cid:9) T t =1 , ξ
[Table]
, n )
Update V it with
Appendix C) for
Stochastic Dual Dynamic Programming (SDDP).
Given the problem speciﬁcation (3) we now consider solution strategies.
Stochastic dual dynamic programming (SDDP) (Shapiro et al., 2014; F ¨ ullner & Rebennack, 2021) is a state-of-the-art approach that exploits the key observation in Theorem 1 that the optimal V function can be expressed as a maximum over a ﬁnite number of linear components.
Given this insight, SDDP applies Bender’s decomposition to the sample averaged approximation of (3).
In particular, it performs two steps in each iteration: (i) in a forward pass from t = 0 , trial solutions for each stage are generated by solving subproblems (4) using the current estimate of the future expected-cost-to-go function V t +1 ; (ii) in a backward pass from t =
T , each of the V -functions are then updated by adding cutting planes derived from the optimal actions x t − 1 obtained in the forward pass.
(Details for the cutting plan derivation and the connection to TD-learning are given in Appendix C due to lack of space.)
After each iteration, the current V t +1 provides a lower bound on the true optimal expected-cost-to-go function, which is being successively tightened; see Algorithm 1.
MSSO vs. MDPs.
At the ﬁrst glance, the dynamics in MSSO (3) is describing markovian relationship on actions , i.e. , the current action x t is determined by current state ξ t and previous action x t − 1 , which is different from the MDPs on markovian states .
However, we can equivalently reformulate MSSO as a MDP with state-dependent feasible action set, by deﬁning the t -th step state as s t :=
( x t − 1 , ξ t ) , and action as a t := x t ∈ χ t ( x t − 1 , ξ t ) .
This indeed leads to the markovian transition p set ( s t +1 | s t , x t )
= 1 ( x t ∈ χ t ( x t − 1 , ξ t )) p t +1 ( ξ t +1 ) , where 1 ( x t ∈ χ t ( x t − 1 , ξ t )) :
= { 1 if x t ∈ χ t ( x t − 1 , ξ t ) , 0 otherwise } .
Although we can represent MSSO equivalently in MDP, the MDP formulation introduces extra difﬁculty in maintaining state-dependent feasible action and ignores the linear structure in feasibility, which may lead to the inefﬁciency and infeasibility when applying RL algorithms (see Section 5).
Instead MSSO take these into consideration, especially the feasibility.
Unfortunately, MSSO and MDP comes from different communities, the notational conventions of the
MSSO versus MDP literature are directly contradictory : the Q -function in (4) corresponds to the state-value V -function in the MDP literature, whereas the V -function in (4) is particular to the MSSO setting, integrating out of randomness in state-value function, which has no standard correspondent in the MDP literature.
In this paper, we will adopt the notational convention of MSSO .
N EURAL S TOCHASTIC D UAL D YNAMIC P ROGRAMMING
Although SDDP is a state-of-the-art approach that is widely deployed in practice, it does not scale well in the dimensionality of the action space (Lan, 2020).
That is, as the number of decision variables in x t increases, the number of generated cutting planes in the V t +1 approximations tends to grow exponentially, which severely limits the size of problem instance that can be practically solved.
To overcome this limitation, we develop a new approach to scaling up SDDP by leveraging the generalization ability of deep neural networks across different MSSO instances in this section.
We ﬁrst formalize the learning task by introducing the contextual MSSO .
Speciﬁcally, as discussed in Section 2, the problem context U
= { u t } Tt =1 with u t :
=
( P t , c t , A t , B t , b t ) soly deﬁnes the
MSSO problem, therefore, we denote W ( U ) as an instance of MSSO (3) with explicit dependence on U .
We assume the MSSO samples can be instantiated from contextual MSSO following some distribution, i.e. , W ( U ) ∼ P ( W ) , or equivalently, U ∼ P ( U ) .
Then, instead of treating each MSSO independently from scratch, we can learn to amortize and generalize the optimization across different
MSSOs in P ( W ) .
We develop a meta-learning strategy where a model is trained to map the u t and t to a piecewise linear convex V t -approximator that can be directly used to initialize the SDDP solver in Algorithm 1.
In principle, if optimal value information can be successfully transferred between similar problem contexts, then the immense computation expended to recover the optimal V t functions for previous problem contexts can be leveraged to shortcut the nearly identical computation of the optimal V t functions for a novel but similar problem context.
In fact, as we will demonstrate below, such a transfer strategy proves to be remarkably effective.
3. N EURAL A RCHITECTURE FOR M APPING TO V ALUE F UNCTION R EPRESENTATIONS
To begin the speciﬁc development, we consider the structure of the value functions, which are desired in the deep neural approximator.
• Small approximation error and easy optimization over action: recall from Theorem 1 that the optimal V t -function must be a convex, piecewise linear function.
Therefore, it is sufﬁcient for the output representation from the deep neural model to express max -afﬁne function approximations for V t , which conveniently are also directly usable in the minimization (6) of the SDDP solver.
• Encode the instance-dependent information: to ensure the learned neural mapping can account for instance speciﬁc structure when transferring between tasks, the output representation needs to encode the problem context information, { ( P t , c t , A t , B t , b t ) }
Tt =1 .
• Low-dimension representation of state and action: the complexity of subproblems in SDDP depends on the dimension of the state and action exponentially, therefore, the output V t -function approximations should only depend on a low-dimension representation of x .
For the ﬁrst two requirements, we consider a deep neural representation for functions f ( · , u t )
∈ M K for t = 1 , . . .
, T , where M K is the piece-wise function class with K components, i.e. ,
[Equation]
That is, such a function f takes the problem context information u as input, and outputs a set of parameters ( { α k } , { β k } ) that deﬁne a max -afﬁne function φ .
We emphasize that although we consider M K with ﬁxed number of linear components, it is straightforward to generalize to the function class with context-dependent number of components via introducing learnable K ( u t ) ∈ N .
A key property of this output representation is that it always remains within the set of valid Vfunctions, therefore, it can be naturally incorporated into SDDP as a warm start to reﬁne the solution.
This approach leaves design ﬂexibility around the featurization of the problem context U and actions x , while enabling the use of neural networks for f ( · , u ) , which can be trained end-to-end.
To further achieve a low-dimensional dependence on the action space while retaining convexity of the value function representations for the third desideratum, we incorporate a linear projection x =
Gy with y ∈ R p and p <
d , such that G = ψ ( u ) satisﬁes G (cid:62) G
= I .
With this constraint, the mapping f ( · , u ) will be in:
[Equation]
We postpone the learning of f and ψ to Section 3.2 and ﬁrst illustrate the accelerated SDDP solver in the learned effective dimension of the action space in Algorithm 2.
Note that after we obtain the solution y in line 3 in Algorithm 2 of the projected problem, we can recover x =
Gy as a coarse solution for fast inference.
If one wanted a more reﬁned solution, the full SDDP could be run on the un-projected instance starting from the updated algorithm state after the fast call.
Algorithm
) 1: Set G = ψ ( U ) (cid:46) fast inference 2: Projected problem instance { q t } Tt =
= { Gu t } Tt =1 , 3:
[Equation]
˜
[Equation]
t
[Equation]
jt (cid:17)(cid:111) t,j = SDDP (cid:16) { f ( · , q t ) } Tt =1 , ξ 1 , 1 (cid:17)
[Equation]
(cid:46) we only need one forward pass in low-dimension space.
[Equation]
(cid:17)(cid:111) t,j = (cid:110)
G ˜ y t (cid:16) ξ jt (cid:17)(cid:111) t,j , /*
Optional reﬁnement */ 5: (cid:110) x ∗ t ( ξ jt ) (cid:111) t,j = SDDP (cid:16) { f ( · , u t ) }
Tt =1 , ξ 1 , n (cid:17) (cid:46) reﬁne solution.
Practical representation details:
In our implementation, we ﬁrst encode the index of time step t by a positional encoding (Vaswani et al., 2017) and exploit sufﬁcient statistics to encode the distribution P ( ξ ) (assuming in addition that the P t are stationary).
As the functions c t , A t ,
B t and b t are typically static and problem speciﬁc, there structures will remain the same for different P t .
In our paper we focus on the generalization within a problem type ( e.g. , the inventory management) and do not expect the generalization across problems ( e.g. , train on portfolio management and deploy on inventory management).
Hence we can safely ignore these LP speciﬁcations which are typically of high dimensions.
The full set of features are vectorized, concatenated and input to a 2-layer MLP with 512 -hidden relu neurons.
The MLP outputs k linear components, { α k } and { β k } , that form the piece-wise linear convex approximation for V t .
For the projection G , we simply share one G across all tasks, although it is not hard to incorporate a neural network parametrized G .
The overall deep neural architecture is illustrated in Figure 5 in Appendix D.
3. M ETA S ELF -I MPROVED L EARNING
The above architecture will be trained using a meta-learning strategy, where we collect successful prior experience D n := (cid:110)
z i := (cid:16) U, { V ∗
[Equation]
} Tt =
[Equation]
, { x ∗
[Equation]
ξ j ) } T,mt =
[Equation]
,j
[Equation]
i (cid:111) n i =1 by using SDDP to solve a set of training problem instances.
Here, U = { u t } T t =1 denotes a problem context, and { V ∗ t } T t =1 and (cid:8) x ∗ tj (cid:9) T,m t,j are the optimal value functions and actions obtained by SDDP for each stage.
Given the dataset D n , the parameters of f and ψ , W :=
{ W f , W ψ } , can be learned by optimizing the following objective via stochastic gradient descent (SGD):
[Equation]
where G ψ := ψ ( u ) , EMD ( f, V ) denotes the Earth Mover’s Distance between f and V , and σ ( W ) denotes a convex regularizer on W .
Note that the loss function (9) is actually seeking to maximize (cid:80) ni =1 (cid:80) tt
=1 (cid:80) mj
( x i ∗ tj ) (cid:62) G ψ G (cid:62) ψ x i ∗ tj under orthonormality constraints, hence it seeks principle components of the action spaces to achieve dimensionality reduction.
To explain the role of EMD , recall that f ( x, u ) outputs a convex piecewise linear function represented by (cid:8) ( β fk )
(cid:62) x
+ α fk (cid:9) K k =1 , while the optimal value function V
[Equation] [Equation]
in
SDDP is also a convex piecewise linear function, hence expressible by a maximum over afﬁne
1: Initialize dataset D 0 ; 2: for epoch i = 1 , . .
. , n do 3:
Sample a multi-stage stochastic decision problems U = { u t } Tt =1 ∼ P ( U ) ; 4:
Initial (cid:8) V 0 t (cid:9) T t =1 = (1 − γ )
+ γ (cid:110)
f W (cid:16) · , { u t } Tt =1 (cid:17)(cid:111) T t =0 with γ ∼ B ( p i ) ; 5: (cid:18)(cid:110) x ∗ ( ξ jt ) (cid:111) T,m t,j = (cid:19) =
SDDP ( (cid:8) V 0 t (cid:9) T t =1 , ξ 1 , n ) ; 6:
Collect solved optimization instance D i
= D i − 1 ∪ (cid:16) U, { V ∗ t } Tt =1 , { x ∗ t ( ξ j ) } T,mt =1 ,j (cid:17) ; 7: for iter = 1 , . . .
, b do 8:
Sample z l ∼ D i ; 9:
Update parameters W with stochastic gradients: W = W − η ∇ W (cid:96) ( W ; z l ) ; 10: end for 11: end for functions.
Therefore, EMD ( f, V ∗ t ) is used to calculate the distance between the sets (cid:8) β fk , α fk (cid:9) K k
= and { β ∗ l , α ∗ l } tl =1 , which can be recast as min
[Equation]
where D ∈ R K × t denotes the pairwise distances between elements of the two sets.
Due to space limits, please refer to Figure 6 in Appendix D for an illustration of the overall training setup.
The main reason we use EMD is due to the fact that f and V ∗ are order invariant , and EMD provides an optimal transport comparison (Peyr ´ e et al., 2019) in terms of the minimal cost over all pairings.
Remark (Alternative losses): One could argue that it sufﬁces to use the vanilla regression losses, such as the L 2 -square loss (cid:107) f
( · , u, ξ ) − V ∗ ( · , ξ ) (cid:107) 22 , to ﬁt f to V ∗ .
However, there are several drawbacks with such a direct approach.
First, such a loss ignores the inherent structure of the functions.
Second, to calculate the loss, the observations x are required, and the optimal actions from SDDP are not sufﬁcient to achieve a robust solution.
This approach would require an additional sampling strategy that is not clear how to design (Defourny et al., 2012).
Training algorithm: The loss (9) pushes f to approximate the optimal value functions for the training contexts, while also pushing the subspace G ψ to acquire principle components in the action space.
The intent is to achieve an effective approximator for the value function in a low-dimensional space that can be used to warm-start SDDP inference Algorithm 2.
Ideally, this should result in an efﬁcient optimization procedure with fewer optimization variables that can solve a problem instance with fewer forward-backward passes.
In an on-line deployment, the learned components, f and ψ , can be continually improved from the results of previous solves.
One can also optinally exploit the learned component for the initialization of value function in SDDP by annealing with a mixture of zero function, where the weight is sampled from a Bernolli distribution.
Overall, this leads to the
Meta Self-Improved SDDP algorithm, ν - SDDP , shown in Algorithm 3.
R ELATED W ORK
The importance of MSSO and the inherent difﬁculty of solving MSSO problems at a practical scale has motivated research on hand-designed approximation algorithms, as discussed in Appendix A.
Learning-based MSSO approximations have attracted more attentions.
Rachev & R¨omisch (2002); Høyland et al.
; Hochreiter & Pﬂug (2007) learn a sampler for generating a small scenario tree while preserving statistical properties.
Recent advances in RL is also exploited.
Defourny et al.
(2012) imitate a parametrized policy that maps from scenarios to actions from some SDDP solvers.
Direct policy improvement from RL have also been considered.
Ban & Rudin (2019) parametrize a policy as a linear model in (25), but introducing large approximation errors.
As an extension, Bertsimas & Kallus (2020); Oroojlooyjadid et al.
(2020) consider more complex function approximators for the policy parameterization in (25).
Oroojlooyjadid et al.
; Hubbs et al.
Balaji et al.
(2019); Barat et al.
(2019) directly apply deep RL methods.
Avila et al.
(2021) exploit off-policy
RL tricks for accelerating the SDDP Q -update.
More detailed discussion about Avila et al.
(2021) can be found in Appendix A. Overall, the majority of methods are not able to easily balance MSSO problem structures and ﬂexibility while maintaining strict feasibility with efﬁcient computation.
They also tend to focus on learning a policy for a single problem, which does not necessarily guarantee effective generalization to new cases, as we ﬁnd in the empirical evaluation.
Context-based meta-RL is also relevant, where the context-dependent policy (Hausman et al., 2018;
Rakelly et al., 2019; Lan et al., 2019) or context-dependent value function (Fakoor et al., 2019;
Arnekvist et al., 2019; Raileanu et al., 2020) is introduced.
Besides the difference in MSSO vs.
MDP in Section 2, the most signiﬁcant difference is the parameterization and inference usage of context-dependent component.
In ν -SDDP, we design the speciﬁc neural architecture with the output as a piece-wise linear function, which takes the structure of MSSO into account and can be seamlessly integrated with SDDP solvers for further solution reﬁnement with the feasibility guaranteed ; while in the vanilla context-based meta-RL methods, the context-dependent component with arbitrary neural architectures, which will induce extra approximation error, and is unable to handle the constraints.
Meanwhile, the design of the neural component in ν -SDDP also leads to our particular learning objective and stochastic algorithm, which exploits the inherent piece-wise linear structure of the functions, meanwhile bypasses the additional sampling strategy required for alternatives.
E XPERIMENTS
[Table]
Problem Deﬁnition.
We ﬁrst tested on inventory optimization with the problem conﬁguration in Table.
We break the problem contexts into two sets: 1) topology , parameterized via the number of suppliers S , inventories I , and customers C ; 2) decision horizon T .
Note that in the Mid-Lng setting there are 310 continuous action variables, which is of magnitudes larger than the ones used in inventory control literature (Graves & Willems, 2008) and benchmarks, e.g. , ORL (Balaji et al., 2019) or meta-RL (Rakelly et al., 2019) on MuJoCo (Todorov et al., 2012).
Within each problem setting, a problem instance is further captured by the problem context.
In inventory optimization, a forecast model is ususally used to produce continuous demand forecasts and requires re-optimization of the inventory decisions based on the new distribution of the demand forecast, forming a group of closely related problem instances.
We treat the parameters of the demand forecast as the primary problem context.
In the experiment, demand forecasts are synthetically generated from a normal distribution: d t ∼ N ( µ d , σ d ) .
For both problem settings, the mean and the standard deviation of the demand distribution are sampled from the meta uniform distributions: µ d ∼ U (11 , 20) , σ d ∼ U (0 , 5) .
Transport costs from inventories to customers are also subject to frequent changes.
We model it via a normal distribution: c t ∼ N ( µ c , σ c ) and use the distribution mean µ c ∼ U (0 . 3 , 0 . 7) as the secondary problem context parameter with ﬁxed σ c = 0 .
Thus in this case, the context for each problem instance that f ( · , µ t ) needs to care about is u t = ( µ d , σ d , µ c ) .
The second environment is portfolio optimization .
A forecast model is ususally used to produce updated stock price forcasts and requires re-optimization of asset allocation decisions based on the new distribution of the price forecast, forming a group of closely related problem instances.
We use an autoregressive process of order 2 to learn the price forecast model based on the real daily stock prices in the past 5 years.
The last two-day historical prices are used as problem context parameters in our experiments.
In this case the stock prices of ﬁrst two days are served as the context U for f .
Due to the space limitation, we postpone the detailed description of problems and additional performances comparison in Appendix E.
Baselines.
In the following experiments, we compare ν -SDDP with mainstream methodologies:
• SDDP-optimal: This is the SDDP solver that runs on each test problem instance until convergence, and is expected to produce the best solution and serve as the ground-truth for comparison.
• SDDP-mean: It is trained once based on the mean values of the problem parameter distribution, and the resulting V -function will be applied in all different test problem instances as a surrogate of the true V -function.
This approach enjoys the fast runtime time during inference, but would yield suboptimal results as it cannot adapt to the change of the problem contexts.
• Model-free RL algorithms: Four RL algorithms, including DQN, DDPG, SAC, PPO, are directly trained online on the test instances without the budget limit of number of samples.
So this setup has more privileges compared to typical meta-RL settings.
We only report the best RL result in Table 2 and Table 3 due to the space limit.
Detailed hyperparameter tuning along with the other performance results are reported in Appendix E.
[Table] [Table]
• ν -SDDP-fast: This is our algorithm where the the meta-trained neural-based V -function is directly evaluated on each problem instance, which corresponds to Algorithm 2 without the last reﬁnement step.
In this case, only one forward pass of SDDP using the neural network predicted V -function is needed and the V -function will not be updated.
The only overhead compared to SDDP-mean is the feed-forward time of neural network, which can be ignored compared to the expensive LP solving.
• ν -SDDP-accurate:
It is our full algorithm presented in Algorithm 2 where the meta-trained neural-based V -function is further reﬁned with 10 more iterations of vanilla SDDP algorithm.
5. S OLUTION Q UALITY C OMPARISON
For each new problem instance, we evaluate the algorithm performance by solving and evaluating the optimization objective value using the trained V -function model over 50 randomly sampled trajectories.
We record the mean ( candidate ) and the standard derivation of these objective values produced by each candidate method outlined above.
As SDDP-optimal is expected to produce the best solution, we use its mean on each problem instance to normalize the difference in solution quality.
Speciﬁcally, error ratio of method candidate with respect to SDDP-optimal is:
[Equation]
Inventory optimization: We report the average optimalty ratio of each method on the held-out test problem set with 100 instances in Table 2.
By comparison, ν -SDDP learns to adaptive to each problem instance, and thus is able to outperform these baselines by a signiﬁcantly large margin.
Also we show that by tuning the SDDP with the V -function initialized with the neural network generated cutting planes for just 10 more steps, we can further boost the performance ( ν -SDDP-accurate).
In addition, despite the recent reported promising results in applying deep RL algorithms in small-scale inventory optimization problems (Bertsimas & Kallus, 2020; Oroojlooyjadid et al., 2020; 2021;
Hubbs et al., 2020; Balaji et al., 2019; Barat et al., 2019), it seems that these algorithms get worse results than SDDP and ν -SDDP variants when the problem size increases.
We further report the average variance along with its standard deviation of different methods in Table 3.
We ﬁnd that generally our proposed ν -SDDP (both fast and accurate variants) can yield solutions with comparable variance compared to SDDP-optimal.
SDDP-mean gets higher variance, as its performance purely depends on how close the sampled problem parameters are to their means.
Portfolio optimization: We evaluated the same metrics as above.
We train a multi-dimensional second-order autoregressive model for the selected US stocks over last 5 years as the price forecast model, and use either synthetic (low) or estimated (high) variance of the price to test different models.
When the variance is high, the best policy found by SDDP-optimal is to buy (with appropriate but different asset allocations at different days) and hold for each problem instance.
We found our ν -SDDP is able to rediscover this policy; when the variance is low, our model is also able to achieve much lower error ratio than SDDP-mean.
We provide study details in Appendix E.2.
5. T RADE - OFF B ETWEEN R UNNING T IME AND A LGORITHM P ERFORMANCE
We study the trade-off between the runtime and the obtained solution quality in Figure 2 based on the problem instances in the test problem set.
In addition to ν -SDDP-fast and SDDP-mean, we plot the solution quality and its runtime obtained after different number of iterations of SDDP (denoted as
[Figure]
[Figure] [Figure]
SDDP-n with n iterations).
We observe that for the small-scale problem domain, SDDP-mean runs the fastest but with very high variance over the performance.
For the large-scale problem domain, ν -SDDP-fast achieves almost the same runtime as SDDP-mean (which roughly equals to the time for one round of SDDP forward pass).
Also for large instances, SDDP would need to spend 1 ormagnitudes of runtime to match the performance of ν -SDDP-fast.
If we leverage ν -SDDP-accurate to further update the solution in each test problem instance for just 10 iterations, we can further improve the solution quality.
This suggests that our proposed ν -SDDP achieves better time-solution trade-offs.
5. S TUDY OF N UMBER OF G ENERATED C UTTING P LANES
In Figure 3 we show the performance of ν -SDDP-fast with respect to different model capacities, captured by the number of cutting planes the neural network can generate.
A general trend indicates that more generated cutting planes would yield better solution quality.
One exception lies in the
Mid-Lng setting, where increasing the number of cutting planes beyond 64 would yield worse results.
As we use the cutting planes generated by last n iterations of SDDP solving in training ν -SDDP-fast, our hypothesis is that the cutting planes generated by SDDP during the early stages in large problem settings would be of high variance and low-quality, which in turn provides noisy supervision.
A more careful cutting plane pruning during the supervised learning stage would help resolve the problem.
5. L OW -D IMENSION P ROJECT P ERFORMANCE
Finally in Figure 4 we show the performance using low-rank projection.
We believe that in reality customers from the same cluster (e.g., region/job based) would express similar behaviors, thus we created another synthetic environment where the customers form 4 clusters with equal size and thus have the same demand/transportation cost within each cluster.
We can see that as long as the dimension goes above 80, our approach can automatically learn the low-dimension structure, and achieve much better performance than the baseline SDDP-mean.
Given that the original decision problem is in 310-dimensional space, we expect having 310/4 dimensions would be enough, where the experimental results veriﬁed our hypothesis.
We also show the low-dimension projection results for the problems with full-rank structure in Appendix E.1.
C ONCLUSION
We present Neural Stochastic Dual Dynamic Programming , which learns a neural-based V -function and dimension reduction projection from the experiences of solving similar multi-stage stochastic optimization problems.
We carefully designed the parametrization of V -function and the learning objectives, so that it can exploit the property of MSSO and be seamlessly incorporated into SDDP.
Experiments show that ν -SDDP signiﬁcantly improves the solution time without sacriﬁcing solution quality in high-dimensional and long-horizon problems. A M ORE R ELATED W ORK
To scale up the MSSO solvers, a variety of hand-designed approximation schemes have been investigated.
One natural approach is restricting the size of the scenario tree using either a scenariowise or state-wise simpliﬁcation.
For example, as a scenario-wise approach, the expected value of perfect information (EVPI, (Birge, 1982; Hentenryck & Bent, 2006)) has been investigated for optimizing decision sequences within a scenario, which are then heuristically combined to form a full solution.
Bareilles et al. (2020) instantiates EVPI by considering randomly selected scenarios in a progressive hedging algorithm (Rockafellar & Wets, 1991) with consensus combination. For a stage-wise approach, a two-stage model can be used as a surrogate, leveraging a bound on the approximation gap (Huang & Ahmed, 2009). All of these approximations rely on a ﬁxed prior design for the reduction mechanism, and cannot adapt to a particular distribution of problem instances.
Consequently, we do not expect such methods to be competitive with learning approaches that can adapt the approximation strategy to a given problem distribution.
Difference to Learning from Cuts in Avila et al.
(2021): the Batch Learning-SDDP (BL-SDDP) is released recently, where machine learning technique is also used for accelerating MSSO solver.
However, this work is signiﬁcant different from the proposed ν -SDDP:
• Firstly and most importantly, the setting and target of these works are orthogonal: the BL-SDDP speeds up the SDDP for a particular given MSSO problem via parallel computation; while ν -SDDP works for the meta-learning setting that learns from a dataset composed by plenty of MSSO problems sampled from a distribution, and the learning target is to generalize to new MSSO instances from the same distribution well; • The technique contribution in BL-SDDP and ν -SDDP are different.
Speciﬁcally, BL-SDDP exploits existing off-policy RL tricks for accelerating the SDDP Q -update; while we proposed two key techniques for quick initialization i) , with predicted convex functions; ii) , dimension reduction techniques, to generalize different MSSOs and alleviate curse-
of-dimension issues in SDDP, which has not been explored in BL-SDDP.
Despite being orthogonal, we think that the BL-SDDP can be used in our framework to provide better supervision for our cut function prediction, and serve as an alternative for ﬁne-tuning after ν -SDDP-fast.
One potential drawback of our ν -SDDP is that, when the test instance distributions deviate a lot from what has been trained on, the neural initialization may predict cutting planes that are far away from the good ones, which may slow down the convergence of SDDP learning.
Characterizing in-distribution v.s. out-of-distribution generalization and building the conﬁdence measure is an important future work of current approach.
B P RACTICAL P ROBLEM I NSTANTIATION
In this section, we reformulate the inventory control and portfolio management as multi-stage stochastic decision problems.
B. I NVENTORY C ONTROL
Let S, V, C be the number of suppliers, inventories, and customers, respectively.
We denote the parameters of the inventory control optimization as:
• procurement price matrix: p t ∈ R SV × 1 ; • sales price matrix: q t ∈ R V C × 1 ; • unit holding cost vector: h t ∈ R V × 1 ; • demand vector: d t ∈ R C × 1 ; • supplier capacity vector: u t ∈ R S × 1 ; • inventory capacity vector: v t ∈ R V × 1 ; • initial inventory vector w 0 ∈ R V × 1 .
The decision variables of the inventory control optimization are denoted as:
• sales variable: y t ∈ R V C × 1 , indicating the amount of sales from inventories to customers at the beginning of stage t ; • procurement variable: z t ∈ R SV × 1 , indicating the amount of procurement at the beginning of stage t after sales; • inventory variable: w t ∈ R V × 1 , indicating the inventory level at the beginning of stage t after procurement.
We denote the decision variables as
[Equation]
the state as
[Equation]
The goal of inventory management is to maximize the net proﬁt for each stage, i.e. ,
[Equation]
and subject to the constraints of 1) supplier capacity; 2) inventory capacity; 3) customer demand, i.e. ,
[Equation]
x t [Equation] [Equation] (demand
[Equation]
[Equation]
(supplier
[Equation]
[Equation]
(inventory capacity
[Equation]
[Equation]
by inventory
[Equation]
[Equation]
(inventory transition) (16)
[Equation]
To sum up, the optimization problem can be deﬁned recursively as follows:
[Equation]
In fact, the inventory control problem (18) is simpliﬁed the multi-stage stochastic decision problem (3) by considering state independent transition.
B. P ORTFOLIO M ANAGEMENT
Let I be the number of assets, e.g. , stocks, being managed.
We denote the parameters of the portfolio optimization are:
• ask (price to pay for buying) open price vector p t ∈
R
I × 1 ; • bid (price to pay for sales) open price vector q t ∈ R
I × 1 ; • initial amount of investment vector w
∈
R
I × 1 ; • initial amount of cash r 0 ∈ R + .
The decision variables of the portfolio optimization are:
• sales vector y t ∈ R I × 1 , indicating the amount of sales of asset i at the beginning of stage t .
• purchase vector z t ∈ R I × 1 , indicating the amount of procurement at the beginning of stage t ; • holding vector w t ∈
R
I × 1 , indicating the amount of assets at the beginning of stage t after purchase and sales; • cash scalar r t , indicating the amount of cash at the beginning of stage t .
We denote the decision variables as
[Equation]
and the state as
[Equation]
The goal of portfolio optimization is to maximize the net proﬁt i.e. ,
[Equation]
subject to the constraints of initial investment and the market prices, i.e. ,
[Equation]
With the c t and χ t ( x t − 1 , ξ t ) deﬁned above, we initiate the multi-stage stochastic decision problem (3) for portfolio management.
C D ETAILS ON S TOCHASTIC D UAL D YNAMIC P ROGRAMMING
We have introduced the SDDP in Section 2.
In this section, we provide the derivation of the updates in forward and backward pass,
• Forward pass , updating the action according to (5) based on the current estimation of the value function at each stage via (4).
Speciﬁcally, for i -th iteration of t -stage with sample ξ jt , we solve the optimization x
[Equation]
In fact, the V -function is a convex piece-wise function.
Speciﬁcally, for i -th iteration of t -stage, we have
[Equation]
Then, we can rewrite the optimization (24) into standard linear programming, i.e. ,
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
calculate
[Equation]
Then, we have the
[Equation]
which is still convex piece-wise linear function, with
[Equation]
where
[Equation]
with ( ω t ( ξ t ) , ρ t ( ξ t )) as the optimal dual solution with realization ξ t .
In fact, although we split the forward and backward pass, in our implementation, we exploit the primal-dual method for LP, which provides both optimal primal and dual variables, saving the computation cost.
Note that SDDP can be interpreted as a form of TD-learning using a non-parametric piecewise linear model for the V -function.
It exploits the property induced by the parametrization of value functions, leading to the update w.r.t.
V -function via adding dual component by exploiting the piecewise linear structure in a closed-form functional update.
That is, TD-learning (Sutton & Barto, 2018; Bertsekas, 2001) essentially conducts stochastic approximate dynamic programming based on the Bellman recursion (Bellman, 1957).
D N EURAL N ETWORK AND L EARNING S YSTEM D ESIGN
Neural network design: In Figure 5 we present the design of the neural network that tries to approximate the V -function.
The neural network takes two components as input, namely the feature vector that represents the problem conﬁguration, and the integer that represents the current stage of the multi-stage solving process.
The stage index is converted into ‘time-encoding‘, which is a 128-dimensional learnable vector.
We use the parameters of distributions as the feature of φ ( P ( ξ )) for simplicity.
The characteristic function or kernel embedding of distribution can be also used here.
The feature vector will also be projected to the same dimension and added together with the time encoding to form as the input to the MLP.
The output of MLP is a matrix of size k × ( N + 1) , where k is the number of linear pieces, N is the number of variable (and we also need one additional dimension for intercept).
The result is a piecewise linear function that speciﬁes a convex lowerbound.
We show an illustration of 2D case on the right part of the Figure 5.
This neural network architecture is expected to adapt to different problem conﬁgurations and time steps, so as to have the generalization and transferability ability across different problem conﬁgurations.
Remark (Stochastic gradient computation): Aside from G ψ , unbiased gradient estimates for all other variables in the loss (9) can be recovered straightforwardly.
However, G ψ requires special treatment since we would like it to satisfy the constraints G (cid:62) ψ G ψ
= I p .
Penalty method is one of the choices, which switches the constraints to a penalty in objective, i.e. , η (cid:13)(cid:13)(cid:13) G (cid:62) ψ G ψ − I p (cid:13)(cid:13)(cid:13) 2 2 .
However, the solution satisﬁes the constraints, only if η → ∞ (Nocedal & Wright, 2006, Chapter 17). We [Figure]
[Figure] [Figure] derive the gradient over the Stiefel manifold (Mahony et al., 1996), which ensures the orthonormal constraints,
[Equation]
34) with Ξ := (cid:80) t (cid:80) ni
=1 (cid:80)
mj x i ∗ tj (cid:0) x i ∗ tj (cid:1) (cid:62) .
Note that this gradient can be estimated stochastically since Ξ can be recognized as an expectation over samples.
The gradients on Stiefel manifold (cid:8) G | G (cid:62) G
= I (cid:9) can be found in Mahony et al.
We derive the gradient (34) via Lagrangian for self-completeness, following Xie et al.
Consider the Lagrangian as
[Equation]
, where the Λ is the Lagrangian multiplier.
Then, the gradient of the Lagrangian w.r.t.
G ψ is
[Equation]
[Equation]
With the optimality condition
[Equation]
[Equation]
Plug (36) into the gradient (35), we have the optimality condition,
[Equation]
To better numerical isolation of the individual eigenvectors, we can exploit Gram-Schmidt process into the gradient estimator (34), which leads to the generalized Hebbian rule (Sanger, 1989;
Kim et al., 2005; Xie et al., 2015), (cid:94)
[Equation]
[Equation] The LT ( · ) extracts the lower triangular part of a matrix, setting the upper triangular part and diagonal to zero, therefore, is mimicking the Gram-Schmidt process to subtracts the contributions from each [Figure] other eigenvectors to achieve orthonormality.
Sanger (1989) shows that the updates with (38) will converges to the ﬁrst p eigenvectors of Ξ .
System design: Next we present the entire system end-to-end in Figure 6.
• Task sampling : the task sampling component draws the tasks from the same meta distribution.
Note that each task is a speciﬁcation of the distribution ( e.g. , the center of Gaussian distribution), where the speciﬁcation follows the same meta distribution.
• We split the task instances into train, validation and test splits: – Train: We solve each task instance using SDDP.
During the solving of SDDP we need to perform multiple rounds of forward pass and backward pass to update the cutting planes ( V -functions), as well as sampling trajectories for monte-carlo approximation.
The learned neural V -function will be used as initialization.
After SDDP solving converges, we collect the corresponding task instance speciﬁcation (parameters) and the resulting cutting planes at each stage to serve as the training supervision for our neural network module.
– Validation: We do the same thing for validation tasks, and during training of neural network we will dump the models that have the best validation loss.
– Test:
In the test stage, we also solve the SDDP until convergence as groundtruth, which is only used for evaluating the quality of different algorithms.
For our neural network approach, we can generate the convex lower-bound using the trained neural nework, conditioning on each pair of (test task instance speciﬁcation, stage index).
With the predicted V -functions, we can run the forward pass only once to retrieve the solution at each stage.
Finally we can evaluate the quality of the obtained solution with respect to the optimal ones obtained by SDDP.
E M ORE E XPERIMENTS E. I NVENTORY O PTIMIZATION E.1. A DDITIONAL R ESULTS ON ν -SDDP
We ﬁrst show the full results of time-solution quality trade-off in Figure 7, and how ν -SDDP-accurate improves from ν -SDDP-fast with better trade-off than SDDP solver iterations in Figure 8.
We can see the conclution holds for all the settings, where our proposed ν -SDDP achieves better trade-off.
Then we also show the ablation results of using different number of predicted cutting planes in
Figure 9.
We can see in all settings, generally the more the cutting planes the better the results.
This suggests that in higher dimensional case it might be harder to obtain high quality cutting planes, and [Figure] [Figure] due to the convex-lowerbound nature of the V -function, having a bad cutting plane could possibly hurt the overall quality.
How to prune and prioritize the cutting planes will be an important direction for future works.
We provide the full ablation results of doing low-dimensional projection for solving SDDP in Figure 10.
The trend generally agrees with our expectation that, there is a trade-off of the lowdimensionality that would balance the quality of LP solving and the difﬁculty of neural network learning.
Longer horizon: we further experiment with the Mid-Lng-joint ( µ d & σ d & µ c ) by varying T in { 10 , 20 , 30 , 40 , 50 } .
See Table 4 for more information.
[Table]
E.1. A DDITIONAL R ESULTS ON M ODEL -F REE RL A LGORITHMS
We implemented the inventory control problem as an environment in the Tensorﬂow TF-Agents library and used the implementation of DQN (Mnih et al., 2013) 1 , DDPG (Lillicrap et al., 2016),
PPO (Schulman et al., 2017) and SAC (Haarnoja et al., 2018) from the TF-Agent to evaluate the performance of these four model-free RL algorithms.
Note that the TF-Agent environment follows a
[Figure]
[Figure]
MDP formulation.
When adapting to the inventory control problem, the states of the environment are the levels of the inventories and the actions are the amount of procurement and sales.
As a comparison, the states and actions in the MDP formulation collectively form the decision varialbles in the MSSO formulation and their relationship – inventory level transition, is captured as a linear constraint.
Following Schulman (2016), we included the timestep into state in these RL algorithms to have non-stationary policies for ﬁnite-horizon problems.
In the experiments, input parameters for each problem domain and instances are normalized for model training for policy gradient algorithms and the results are scaled back for reporting.
We report the average error ratio of these four RL algorithms in Table.
6 along with the average variance in Table.
Note that the SAC performance is also reported in the main text in Table.
2 and
Table.
The model used in the evaluation is selected based on the best mean return over thetrajectories from the validation environment, based on which the hyperparameters are also tuned.
We report the selected hyperparameters for each algorithm in Table.
We use MLP with 3 layers as the
Q -network for DQN, as the actor network and the critic/value network for SAC, PPO and DDPG.
All networks have the same learning rate and with a dropout parameter as 0.001.
[Table]
We see that SAC performs the best among the four algorithms in terms of solution quality.
All the algorithms can not scale to Mid-Long setting.
DDPG, for example, produces a trivial policy of no action in most of setups (thus has an error ratio of 100 ).
The policies learned by DQN and PPO are even worse, producing negative returns 2 .
To understand the behavior of each RL algorithm, we plotted the convergence of the average mean returns in Figure.
11 for the Sml-Sht task.
In each plot, we show four runs of the respective algorithm under the selected hparameter.
We could see that though SAC converges the slowest, it is able to [Table] [Figure]
[Table] [Figure] achieve the best return. For all algorithms, their performance is very sensitive to initialization.
DDPG, for example, has three runs with 0 return, while one run with a return of 1.2.
For PPO and DQN, the average mean returns are both negative .
We further check the performance of these algorithms in the validation environment based on the same problem instance ( i.e. , the same problem parameters) as in SDDP-mean, where the model is trained and selected.
We expect this would give the performance upper bound for these algorithms.
Again similar results are observed.
The best return mean over the validation environment is − 38 .for
PPO, 0 .
95 for DQN, 1 . 31 for SAC and 1 .
41 for DDPG, while the SDDP optimal return value is 1 .
It is also worth noting that DDPG shows the strongest sensitivity to initialization and its performance drops quickly when the problem domain scales up.
E. P ORTFOLIO O PTIMIZATION
We use the daily opening prices from selected Nasdaq stocks in our case study.
This implies that the asset allocation and rebalancing in our portfolio optimization is performed once at each stock market opening day.
We ﬁrst learn a probabilistic forecasting model from the historical prices ranging from 2015-1-1 to 2020-01-01.
Then the forecasted trajectories are sampled from the model for the stochastic optimization.
Since the ask price is always slightly higher than the bid price, at most one of the buying or selling operation will be performed for each stock, but not both, on a given day.
E.2. S TOCK P RICE F ORECAST
The original model for portfolio management in Appendix B.2 is too restrict.
We generalize the model with autoregressive process (AR) of order o with independent noise is used to model and predict the stock price:
[Equation]
where φ i is the autoregressive coefﬁcient and (cid:15) ri ∼ N (0 , σ 2 i ) is a white noise of order i .
(cid:15) o ∼
N (0 , σ 2 o ) is a white noise of the observation.
Each noise term is (cid:15) assumed to be independent.
It is easy to check that the MSSO formulation for portfolio management is still valid by replacing the expectation for (cid:15) , and setting the concatenate state [ p t − o t , q t ] , where p t − o t :=
[ p t − i ] oi =0 .
We use variational inference to ﬁt the model.
A variational loss function (i.e., the negative evidence lower bound (ELBO)) is minimized to ﬁt the approximate posterior distributions for the above parameters.
Then we use the posterior samples as inputs for forecasting.
In our study, we have studied the forecasting performance with different length of history, different orders of the AR process and different groups of stocks.
Figure.
12 shows the ELBO loss convergence behavior under different setups.
As we can see, AR models with lower orders converge faster and smoother. [Figure]
[Figure]
We further compare the forecasting performance with different AR orders.
Figure.
13 plots a sideby-side comparison of the forecasted mean trajectory with a conﬁdence interval of two standard deviations (95%) for 5 randomly selected stocks (with tickers GOOG, COKE, LOW, JPM, BBBY) with AR order of 2 and 5 from 2016-12-27 for 5 trading days.
As we could see, a higher AR order (5) provides more time-variation in forecast and closer match to the ground truth.
In addition, we cluster the stocks based on their price time series (i.e., each stock is represented by a T -dimensional vector in the clustering algorithm, where T is the number of days in the study).
We randomly selected 1000 stocks from Nasdaq which are active from 2019-1-1 to 2020-1-1 and performed k -means clustering to form 8 clusters of stocks.
We take the cluster center time series as the training input.
Figure.
12(c) shows the ELBO loss convergence of an AR process of order 5 based on these 8 cluster center time series.
As we see, the stock cluster time series converge smoother and faster compared with the individual stocks as the aggregated stock series are less ﬂuctuated.
The forecasting trajectories of these 8 stock clusters starting from 2019-03-14 are plotted in Figure.
E.2. S OLUTION Q UALITY C OMPARISON
[Table]
With an AR forecast model of order o , the problem context of a protfolio optimizaton instance is then captured by the joint distribution of the historical stock prices over a window of o days.
We learn this distribution using kernel density estimation.
To sample the scenarios for each stage using the
AR model during training for both SDDP and ν -SDDP, we randomly select the observations from the previous stage to seed the observation sampling in the next stage.
Also we approximate the state representation by dropping the term of p t − o t .
With such treatments, we can obtain SDDP results with manageable computation cost.
We compare the performance of SDDP-optimal, SDDP-mean and ν -SDDP-zeroshot
under different forecasting models for a horizon of 5 trading days.
First we observe that using the AR model with order 2 as the forecasting model, as suggested by the work (Dantzig & Infanger, 1993), produce a very simple forecasting distribution where the mean is monotonic over the forecasting horizon.
As a result, all the algorithms will lead to a simple “buy and hold” policy which make no difference in solution quality.
We further increase the AR order gradually from 2 to 5 and ﬁnd AR order at 5 produces sufﬁcient time variation that better ﬁts the ground truth.
Second we observe that the variance learned from the real stock data based on variational inference is signiﬁcant.
With high variance, both SDDP-optimal and ν -SDDP-zeroshot would achieve the similar result, which is obtained by a similar “buy and hold” policy.
To make the task more challenging, we rescale the standard deviation (STD) of stock price by a factor in Table 8 for both the 5-stock case and 1000-stock case with 8 clusters situations.
For the cluster case, the ν -SDDP-zeroshot can achieve almost the same performance as the SDDP-optimal.
E. C OMPUTATION RESOURCE
For the SDDP algorithms we run using multi-core CPUs, where the LP solving can be parallelized at each stage.
For RL based approaches and our ν -SDDP, we train using a single V100 GPU for each hparameter conﬁguration for at most 1 day or till convergence.