Neural-Progressive Hedging: Enforcing Constraints in Reinforcement Learning with Stochastic Programming
Abstract
We propose a framework, called neural-progressive hedging (NP), that leverages stochastic programming during the online phase of executing a reinforcement learning (RL) policy.
The goal is to ensure feasibility with respect to constraints and risk-based objectives such as conditional value-atrisk (CVaR) during the execution of the policy, using probabilistic models of the state transitions to guide policy adjustments.
The framework is particularly amenable to the class of sequential resource allocation problems since feasibility with respect to typical resource constraints cannot be enforced in a scalable manner.
The NP framework provides an alternative that adds modest overhead during the online phase.
Experimental results demonstrate the efﬁcacy of the NP framework on two continuous real-world tasks: (i) the portfolio optimization problem with liquidity constraints for ﬁnancial planning, characterized by non-stationary state distributions; and (ii) the dynamic repositioning problem in bike sharing systems, that embodies the class of supply-demand matching problems.
We show that the NP framework produces policies that are better than deep RL and other baseline approaches, adapting to non-stationarity, whilst satisfying structural constraints and accommodating risk measures in the resulting policies.
Additional beneﬁts of the NP framework are ease of implementation and better explainability of the policies.
INTRODUCTION
Reinforcement learning (RL) experienced a surge in popularity when deep models demonstrated superior performance in game playing with Deep Q-learning Networks (DQN)
[Mnih et al., 2013].
The role of RL was cemented when it was used to beat the reigning Go world champion [Silver et al., 2017].
Improvements to deep RL algorithms have abounded, including RL for continuous state and action spaces, with DDPG [Lillicrap et al., 2015], TRPO [Schulman et al., 2015], and PPO [Schulman et al., 2017].
In spite of these advances, the dominance of RL for real-world problems has lagged.
We believe that this is due to three shortcomings.
First, RL policies cannot enforce business rules, or constraints during policy execution.
Yet, often structural constraints must be respected for a policy to be implementable.
Existing methods, such as constrained policy gradient [Achiam et al., 2017] or “safe" RL methods [Garcıa and
Fernández, 2015] do not prevent constraint violations during policy execution.
Moreover, these methods can often be difﬁcult to train and not scalable to large problems.
Second, there is a natural trade-off between expected reward and risk.
The majority of RL algorithms seek to maximize the expected return.
While there have been RL algorithms that optimize for various risk measures, doing so in a scalable manner and under constraints is still challenging.
Third, the sample inefﬁciency of RL has posed an impediment to solving problems where high-ﬁdelity simulators are not available to generate sufﬁciently large number of sample trajectories.
A hope to overcoming this is through the judicious use of models to explore more sparingly the state and action spaces.
We introduce a framework to address these issues for problems with continuous state and action spaces.
An unconstrained RL policy is ﬁrst trained ofﬂine.
During the online execution phase, a stochastic program (SP) is used to reoptimize the given RL policy under constraints and risk measures over a short-term future trajectory.
Once the next action is chosen, the process repeats in a rolling-horizon fashion using updated state information.
We call this neuralprogressive hedging (NP).
During execution time, the NP method aims to exploit the generalization ability of RL to unseen scenarios that are not experienced during training time, jointly with the ability of SP to exploit models and enforce scenario-dependent constraints as well as incorporating risk measures.
Since the NP framework relies on a model-based online phase, it is most useful in problem settings where closed-form models of state transitions are relatively good approximations to the true state transitions.
Sequential and dynamic resource allocation problems are a key example in which the NP framework excels.
Empirically, we show that the NP method results in policies that offer substantial improvements in reward under various risk measures whilst satisfying hard constraints 1 .
Moreover, we observe that the NP policy, with its more sample-efﬁcient initial RL policy followed by the online ﬁne-tuning phase, is able to outperform the fully-trained (and data-hungry) RL policy.
An additional beneﬁt of the framework is ease of implementation: it can be implemented using existing deep RL algorithms and off-the-shelf optimization packages.
Furthermore, the SP counterpart of the NP method allows easily performing sensitivity analysis on the model parameters and ensures that the business rules or constraints are enforced during online execution of a policy which adds signiﬁcant transparency and explainability to the policy choices in comparison to deep RL.
To that end, the key contributions of the paper are as follows: We deﬁne a novel method, neural-progressive hedging, combining stochastic programming model-based online planning with ofﬂine, deep RL for a continuous policy that satisﬁes hard constraints during execution; 2. We incorporate risk-measures such as CVaR without sacriﬁcing model structure or decomposition algorithm; and 3. We demonstrate the efﬁcacy of the NP method on the class of resource allocation models, including two realworld problems: (i) liquidity-constrained portfolio optimization with a CVaR objective; and (ii) dynamic repositioning in a bike sharing system, where the NP method outperforms deep RL, both constrained and unconstrained as well as other baselines.
PRELIMINARIES
Consider the problem of learning a deterministic policy π : S
→ A in a Markov Decision Process (MDP) given by ( S , A , p, f, γ, T, G ) , with continuous states s ∈ S , continuous actions x ∈ A , transition probability distribution p ( s t +1 | s t , x t ) , cost function f ( s t , x t , s t +1 ) ∈ R , discount factor γ ∈ [0 , 1] , decision horizon T , and constraint set
G .
We allow T = ∞ whenever γ < 1 .
The constraint set
G contains a set of K additional cost functions g 1 . . .
g K where g k ( s t , x t , s t +1 ) ∈ R and constants β 1 . . .
β K ∈ R .
Our constrained MDP setting follows that of Altman [1999], where we aim to solve the following problem: minimize π
[Equation]
Without loss of generality we assume a ﬁxed initial state s 1 .
The expectation E π is taken with respect to randomness induced by the transitions s t +1 ∼ p ( ·| s t , x t ) by taking x t
= π ( s t ) , for all t .
Problem (1) is very challenging to solve for general MDPs with continuous states and actions.
We shall now put this constrained MDP in the context of stochastic programming (SP) from which we borrow many of the algorithmic tools in this work.
The key assumption from SP is that all the randomness or uncertainty in the system comes from external sources.
This decoupling of randomness allows us to employ powerful optimization tools in solving the main problem.
Assume
T is ﬁnite and let ξ 1 . . .
ξ T be random variables such that the next state s t +1 is given by s t
+1 = ˜
p
( s t , x t , ξ t ) where ˜ p is a deterministic function once ξ t is ﬁxed.
We call each realization of ξ = ( ξ 1 . . .
ξ T ) a scenario .
Given a particular scenario ξ 2 , one can ﬁnd the best action sequence in “hindsight” by solving the following problem: minimize x =( x
[Equation]
[Equation]
where we deﬁne
[Equation]
If, for each ξ , the functions ˜ f and ˜ g k for all k are all convex in x , then each scenario sub-problem can be readily solved using existing convex optimization tools.
To simplify notation, we deﬁne the constraint set G ( ξ ) :
= { x | ˜ g k
( x, ξ ) ≤ β k , k = 1 . . .
K } , so problem (2) can be stated simply as minimize x ∈G ( ξ ) ˜
f ( x, ξ ) .
Suppose that one starts with a ﬁnite set Ξ of scenarios, with known probability distribution q ( ξ ) where (cid:80) ξ ∈ Ξ q ( ξ )
= 1 .
One can solve problem (2) for each individual ξ ∈ Ξ to obtain a mapping x ( · ) that provides a solution x ( ξ )
= ( x 1 ( ξ ) . . .
x T ( ξ )) for each ξ ∈ Ξ .
Suppose that the action space A ⊆ R n and | Ξ | = N , then x ( · ) ∈
A N × T ⊆
R N × T × n .
How could we then reconcile the various x t ( ξ ) across all ξ ∈ Ξ , at time t ?
For the resulting solutions to be implementable, one needs to enforce a nonanticipative property which states that x t must only depend on information available at time t .
From an MDP point of view, the state s t captures all observations available up to time t , represented by ξ 1 . . .
ξ t − 1 , and therefore x t must only depend on these if it is to be implementable, i.e., x t ( ξ )
= x t ( ξ 1 , . . . , ξ t − 1 ) and x 1 ( ξ ) must be the same for all ξ .
All solutions x ( · ) that satisfy this nonanticipative property can be expressed as:
[Equation]
We use M to denote the space of all nonanticipative mappings.
Deﬁne an inner product on A N × T by (cid:104) x ( · ) , w ( · ) (cid:105) :
= (cid:80) ξ q ( ξ ) (cid:80) Tt =1 (cid:104) x t ( ξ ) , w t ( ξ ) (cid:105) where (cid:104) x t ( ξ ) , w t ( ξ ) (cid:105) is the standard inner product in R n .
Given any ˆ x ( · ) ∈
A N × T , one can ﬁnd a nonanticipative version x ( · ) =
P M [ˆ x ( · )] where P M is the orthogonal projection onto M given by the conditional expectation x t ( ξ ) =
E ξ | ξ 1 ...ξ t − 1 ˆ x t ( ξ ) for all t and ξ .
Note that P M can be computed via simple averaging over the appropriate subsets of scenarios.
Deﬁne G ⊆
A N × T
such that x ( · ) ∈
G iff x ( ξ ) ∈ G ( ξ ) for all ξ .
We then aim to solve the following global problem:
[Equation]
where E ξ ˜ f
( x ( ξ ) , ξ ) = (cid:80) ξ ∈ Ξ q ( ξ ) ˜ f
( x ( ξ ) , ξ ) .
Without the constraint x ( · ) ∈ M , problem (3) would in fact be separable and could be decomposed into solving individual scenarios as in problem (2).
This problem, however, can still be solved in an iterative manner where each iteration involves solving a slightly modiﬁed version of problem (2) for each scenario.
This “progressive hedging” algorithm by Rockafellar and
Wets [1991], which is an application of the proximal point algorithm, involves keeping track of the solution x i ( · ) as well as a Lagrange multiplier
λ i ( · ) in each iteration i , until convergence.
It also involves a parameter ν i > 0 , which may be constant for all i .
Each iteration involves solving the following steps:
At iteration i , solve the following for each scenario ξ :
[Equation]
Compute x i +1 ( · ) =
P M [ˆ x i ( · )] .
Update the Lagrange multiplier
λ i +1 ( · ) =
λ i ( [Equation] + ν i
[ˆ x i ( · ) − x i +1 ( · )] .
In the case where ˜ f
and G are both convex, the algorithm is guaranteed to converge to an optimal solution x ∗ ( · ) of problem (3) starting from arbitrary x 1 ( · ) and λ 1 ( · ) .
Local convergence to a stationary point for nonconvex ˜ f was shown by Rockafellar [2019].
Initialization: Obtain RL policy π θ .
Deﬁne inner convergence criterion (cid:15) , convex combination parameters κ i and penalty parameters ν i > 0 for i > 0 .
for τ = 1 , 2 , . . .
, do
Observe state s ( τ ) .
Sample scenario set Ξ , and query π θ to obtain x π ( · ) .
Set x 1 ( · ) =
x π ( · ) .
Set λ 1 ( · ) = 0 , u 1 ( · ) =
and i = 1 .
while convergence criterion δ i > (cid:15) do 1.
Solve, for each ξ ∈ Ξ , (5) (or (4) for the riskneutral case) to obtain ˆ x i ( ξ ) and ˆ y i ( ξ ) .
P i multipliers u u end while
From converged solution x ∗ ( · ) , obtain and execute x ∗ 1 .
end for
The SP framework can be adapted to measures of risk.
Consider CVaR, the conditional value-at-risk, a popular measure for ﬁnding risk-averse solutions.
The CVaR of a random variable Z at level α ∈ [0 , 1) can be written as:
[Equation]
CVaR at α = 0 gives the expectation. We solve the CVaR version of problem (3), replacing the expectation E ξ with CVaR α , by following a modiﬁed progressive hedging algorithm [Rockafellar, 2018] with an introduction of an additional variable y i ( ξ ) ∈ R and the corresponding dual u i ( ξ ) ∈ R for each ξ .
Instead of equation (4), we solve equation (5) in step 1 with corresponding changes in steps 2 and 3.
[Equation]
NEURAL-PROGRESSIVE HEDGING
We introduce Neural-Progressive Hedging (NP) method combining the generalization capability of ofﬂine RL with the ability of SP through an online phase to exploit models while enforcing scenario-dependent constraints and risk measures.
The key steps of the NP method are shown compactly in Algorithm 1.
The NP method works as follows: an unconstrained RL policy π θ , parameterized by θ , is obtained by solving (1), or its risk-aware counterpart, without constraints.
In each time-step τ , the NP method observes current state s ( τ ) and queries RL policy π θ to get initial action x π ( · ) .
The new
NP policy is guided by the initial RL policy via a convex combination parameter κ so that, at convergence, the executed actions satisfy constraints of G and the risk measures.
Inner iterations are denoted by i = 1 , . . .
; at each iteration i , the SP sub-problems are solved for each scenario ξ ∈ Ξ with updated Lagrangian multipliers λ i , u i to obtain the dual solution ˆ x i ( ξ ) and ˆ y i ( ξ ) .
Then, we project ˆ x i ( ξ ) onto a feasible space P M [ˆ x i ( · )] , that satisﬁes the nonanticipative property, by averaging over all the scenarios.
The primal solution x i +1 ( · ) is obtained as a convex combination with the initial RL policy x π ( · ) then projected with P M [ˆ x i ( · )] .
We then update multipliers
λ i , u i , and parameters κ i , and
ν i .
This iterative process continues until the difference between primal and dual solutions, δ i , is below a pre-deﬁned threshold (cid:15) .
Resource Allocation Problems: The NP approach is particularly effective for the class of resource allocation problems.
In such applications, the main source of uncertainty is external – consider stock price changes or customer demands – and to a large extent such random variables are unaffected by the actions of the policy.
A scenario generator can hence be readily trained using historical data.
The set of scenarios, Ξ , is obtained by sampling from such a scenario generator.
Given a scenario, ξ , this policy can then be queried at any state s t to obtain the corresponding action x t .
Given a ﬁnite scenario set Ξ , we can obtain from π θ its solution x π ( · ) ∈ M .
THEORETICAL ANALYSIS
The parameter κ i blends the ofﬂine RL policy with the solution from SP (Step 2 in Algorithm 1).
The assumption below covers the settings of warm start, where κ 1 = 1 and ˆ ı = 2 , and imitation learning, where κ i is a decreasing sequence such as (1 + i ) − 2 , where 1 ≤ ˆ ı < ∞ .
Assumption 1 (Imitation learning and warm start) Let κ i → 0 as i → ∞ .
Furthermore, there exists an ˆ ı such that for all i ≥ ˆ ı , κ i = 0 .
Assumption 2 (Existence and local convexity) Assume that the solution set of equation (5) for a CVaR objective, or equation (4) otherwise, is nonempty and ﬁnite, G ( ξ ) is convex and compact, the gradients of ˜ f are locally Lipschitz for each ξ and that the dual penalty parameters
ν i are sufﬁciently large for all i .
Lemma 1 Under Assumption 1, the NP algorithm is equivalent to the progressive hedging algorithm over an inﬁnite number of iterations.
Proof: Assumption 1 states that there exists a ﬁnite iterate ˆ ı such that for all i ≥ ˆ ı , κ i = 0 .
Since x i +1 ( · ) = κ i x π ( · ) + (1 − κ i ) P M
[ˆ x i ( · )] , for all i (cid:48) ≥ ˆ ı , x i (cid:48) ( · ) =
P M [ˆ x i (cid:48) ( · )] , and hence the update of the primal variable of the algorithm reduces to the progressive hedging update.
(cid:4)
Instances of stochastic programming typically make use of discretized support Ξ .
We thus deﬁne the problem (3) in terms of a discrete Ξ and refer to this problem for the remainder of this section.
Assumption 3 (Discrete support) Let Ξ be a discrete support and let 1 . . .
K index each scenario corresponding to a random variable ξ ∈ Ξ , with probability p k = 1 /K .
Then, problem (3) can be expressed as:
[Equation]
Theorem 1 (Convergence of Alg.
1 for Convex ˜ f )
Under Assumptions 1, 2 and 3, along with the convexity of ˜ f , the sequence of iterates ( x i ( · ) , y i ( · ) , λ i ( · ) , u i ( · )) generated by the NP algorithm is such that
[Equation]
[Equation]
u i
[Equation]
u i
[Equation]
, and
[Equation]
with equality at ( x ∗ ( · ) , y ∗ ) in the case of ﬁnite convergence, and thus converges to a local solution ( x ∗ ( · ) , y ∗ ) with ( λ ∗ ( · ) , u ∗ ( · )) as i → ∞ .
Proof: From Lemma 1, Algorithm 1 is equivalent to the Progressive Hedging Algorithm of Rockafellar [2019] when run for an inﬁnite number of iterations.
The convergence of the
Progressive Hedging Algorithm to a solution ( x ∗ ( · ) , y ∗ ( · )) is thus guaranteed under Assumptions 2 and 3 along with the convexity of ˜ f .
(cid:4)
Theorem 2 (Convergence of Alg.
for Nonconvex ˜ f )
Let Assumptions 1, 2 and 3, hold and let ( x i ( · ) , y i ( · )) be a locally optimal solution to each subproblem (5) .
If sequences { x i , y i , λ i , u i } converge to point { x ∗ , y ∗ , λ ∗ , u ∗ } , then ( x ∗ ( · ) , y ∗ ( · )) generated by the NP algorithm is a locally optimal solution to problem (3) .
Proof: From Lemma 1, Algorithm 1 is equivalent to the
Progressive Hedging Algorithm of Rockafellar [2019] when run for an inﬁnite number of iterations.
For nonconvex ˜ f , when the Progressive Hedging Algorithm converges to a point, under Assumptions 2 and 3, it was shown in Rockafellar and Wets [1991] that the point is a stationary point of the problem (3).
(cid:4)
The NP algorithm uses a decomposition of the measurability constraints on the scenario tree from the scenario-speciﬁc constraints, and then proceeds to solve the SP by standard optimization methods.
It should be noted however that the structure and theoretical properties of the NP hold equally with sample average approximation [Bertsimas et al., 2018].
When combining the unconstrained policy x π ( · ) with the constrained solution P M (ˆ x i ( · )) , we also show how the quality of x i +1 evolves as a function of x π ( · ) and P M (ˆ x i ( · )) .
Proposition 1 Let ˜ f be Lipschitz ∀ ξ , i.e., (cid:107) ˜ f
( x ( ξ ) , ξ ) − ˜ f
( x (cid:48) ( ξ ) , ξ ) (cid:107) ≤ L (cid:107) x ( ξ ) − x (cid:48) ( ξ ) (cid:107) .
We have the following bound as a function of κ i and Lipschitz constant L :
[Equation]
Proof: For each scenario ξ , we have
[Equation]
[Table] Naturally, we expect an unconstrained RL solution to achieve a higher objective value, but the executed solution may include constraint violations and excessive risk.
The parameter κ , thus controls the trade-off between a higher objective value and constraint satisfaction and risk aversion.
EXPERIMENTAL RESULTS
To evaluate the performance of the proposed neuralprogressive hedging (NP) approach, we conduct experiments on two real-world domains where risk measures and constraints are an integral part of implementable policies: (i)
Liquidity management through portfolio optimization which seeks to optimally reinvest earnings based on the CVaR whilst maintaining sufﬁcient liquidity; and (ii) Online repositioning which seeks to dynamically match supply-demand when resources (here, represented by bikes in a bike-sharing system) must be continuously rebalanced to meet changes in demand whilst respecting the station capacity constraints.
We compare performance of NP method with Constrained
Policy Optimization (CPO)
[Achiam et al., 2017], and
Lagrangian-relaxed Proximal Policy Optimization (PPOL)
[Ray et al., 2019].
DDPG [Lillicrap et al., 2015] is used to solve the unconstrained RL problems.
Note that when κ = 1 , the NP approach returns the DDPG solution.
Similarly, when κ = 0 , the NP method returns the results of a pure stochastic program (SP), computed using progressive hedging method [Rockafellar, 2019].
Experiment settings: We perform all the experiments on Ubuntu 18.04 virtual machines with 32-core CPU, 64 GB of
RAM, and a single Nvidia Tesla P100 GPU.
The distributed Ray framework and RLlib [Liang et al., 2017] were used for the DDPG method.
The pure SP and NP methods with linear and non-linear objective function are solved using IBM
ILOG CPLEX 12.9 and IPOPT [Wächter and Biegler, 2006], respectively.
The CPO and PPO-L methods are solved using OpenAI safe RL implementation [Ray et al., 2019].
The unconstrained RL policy used as an expert is computed at each time step t using the DDPG algorithm [Lillicrap et al., 2015].
We use a recurrent neural network (RNN) architecture for training the DDPG method with 1 hidden layer consisting of 25 hidden predictor nodes and a tanh nonlinear activation function.
In addition, a long short-term memory (LSTM) model is used to represent the RNN architecture with LSTM cell size 256 and maximum sequence length of 20. Parameter values are as follows: the discounting factor γ = 0 . 99 , minibatch size b =and learning rate lr = 3 e − 5 . Two state-of-the-art methods are used to compare with the constrained RL policy: (i) Constrained policy optimization [Achiam et al., 2017]; and (b) Proximal policy optimization with a Lagrangian penalty [Ray et al., 2019].
For both we use a neural network with 2 hidden layers, each consisting of 256 hidden nodes with tanh nonlinear activation function.
The source codes for the constrained benchmark algorithms can be found at https://github.com/openai/safety-starter-agents.
A discretized scenario tree is used in each decision epoch to solve the NP method for the experiments.
For the ﬁnancial planning example, in each decision period t , we generate a two layer scenario tree where the ﬁrst layer consists of a root node and the second layer includes 1000 nodes, giving rise to 1000 scenarios.
The interest rates for each of the scenarios are sampled from a multi-dimensional log normal distribution whose mean and covariance matrix are estimated from the training data set of price movements in the S&P500.
For the liquidity constraints, we sample 10 liquidity demand processes from a Gaussian distribution with µ = 0 .
025 and σ = 0 .
01 , giving rise to 10,000 scenarios in the second layer of the scenario tree.
For the bike sharing problem, due to its complex non-linear objective function, we generate a two-layer tree with 200 leaf nodes, giving rise to 200 scenarios.
The demand values at stations for each of the scenarios are sampled from a multi-variate normal distribution whose mean and covariance matrix are learnt from 60 days of training demand data [Ghosh et al., 2019].
LIQUIDITY-CONSTRAINED PORTFOLIO OPTIMIZATION
The liquidity management problem seeks to optimally reinvest earnings in a portfolio based on the CVaR whilst maintaining sufﬁcient liquidity.
Too much liquidity means loss of potential returns and too little incurs borrowing costs.
Model-based forecasts of the price movements and liquidity process are generally available in practice.
The overall problem thus involves computing allocations across a universe of ﬁnancial instruments, given observed rewards, prices, and model-based forecasts of the price and liquidity processes.
We have one risk-free liquid instrument.
In each time step a constraint requires that the amount in the liquid account to satisfy forecasted demand.
We consider four portfolios, each with nine stocks and one risk-free instrument.
The state at time t includes the current allocation, observed price changes and liquidity demands up to time t .
The action is a vector, x t = ( x t, 1 , ..., x t,J ) , of allocations across J instruments at time t , where j = 1 is the liquid asset.
Let ξ t, 1 be the cumulative liquidity requirement and
W t the wealth at the beginning of time t .
The constraint set is G ( ξ ) :
= { x | W t · x t, 1 ≥ ξ t, 1 , (cid:80) j x t,j = 1 , t = 1 . . .
T } .
The liquidity requirement (cid:96) t ( ξ ) for time t is sampled from a Gaussian (cid:96) ∼ N ( µ, σ ); µ (cid:96) = 0 .
025 , σ (cid:96) = 0 .
01 and accumulates over time, i.e., ξ t,
1 =
L t − 1 + (cid:96) t , where L t −denotes the accumulated realized liquidity requirement.
We use 11 years of S&P500 daily data from 2009–2019.
The data from 2009–2016 is used for training the unconstrained
RL policy π θ and price movement model.
For the SP and
NP, in each time step, we sample 1000 scenarios from a multi-variate log-normal distribution whose parameters are learnt from the training data.
Hyperparameter tuning of π θ is done using data of 2017–2018.
Tests are done on two consecutive 30 working day periods in 2019 (Jan 1-Feb 11, and Feb 12-Mar 25).
It should be noted that the experiments for these two testing datasets are done independently, where we assume that the initial investment starts with 1 unit of liquid asset at the ﬁrst day.
In Figure 1, we demonstrate the convergence of the neural-progressive hedging method for four asset universes on the ﬁrst testing dataset, in the presence of and after damping to zero the expert guidance after 20 iterations.
Figure 2(a)-(b) shows the mean and standard error in returns of NP with CVaR α = 0 .
95 , 0 . 99 , along with the unconstrained RL policy and the pure SP policy, over four asset universes.
The NP policies with CVaR α = 0 .
95 , 0 .signiﬁcantly outperforms the pure SP policy and improves the average return by 14% and 18% over the DDPG policy.
It should be noted that the variance (demonstrated by the light shaded area) arises from differences in return rates for 4 different asset universes, but our NP method always outperforms other baseline methods for individual asset universe.
Table 1 provides performance metrics including
According to Theorem 1, initial iterates may decrease nonmonotonically but for iterations i ≥ ˆ ı = 20 progression to an optimum is monotonic.
the Sharpe ratio, volatility and maximum daily drawdown (MDD), as well as the performance of four well-known and best performing online portfolio selection algorithms as benchmarks: (i) A uniform constant rebalancing portfolio (uCRP) approach [Cover, 2011]; (ii) Online moving average reversion (OLMAR) [Li and Hoi, 2012]; (iii) Passiveaggressive mean reversion (PAMR) [Li et al., 2012] and (iv) Robust median reversion (RMR)
[Huang et al., 2016].
We use a grid search to optimize the two key hyper-parameters of these online universal portfolio algorithms: namely the lookback window w and threshold parameter (cid:15) 3 .
Average returns and Sharpe ratios of the NP are higher than all the benchmark approaches.
In Figure 2(c), we demonstrate the sample efﬁciency of our expert-guided NP approach.
For this experiment, we train a DDPG policy with fewer samples (referred as “DDPG-LS", “LS=less samples") obtained after 0.5 million training steps, and use it as the expert policy to guide our NP approach. It should be noted that the data for generating scenario samples for the SP counterpart in our NP method is from the 0. million step samples used during training of sub-optimal
DDPG only. Therefore, despite having less training data, NP still provides better returns than the sample-hungry DDPG policy, which is trained to convergence at 1.5 million steps.
A signiﬁcant beneﬁt of the NP framework is the ability to enforce constraints, otherwise difﬁcult to handle in an RL policy.
Figure 3(a)-(b) show the mean and standard error in returns under liquidity constraints.
We compare against a heuristic we call DDPG-H that uses DDPG, but reserves µ (cid:96) + 3 σ (cid:96) of the funds for the 0-interest cash account by re-normalizing the remaining allocations.
DDPG-H thus provides a conservative, but constraint-feasible policy, by construction.
The constrained NP policy outperforms CPO,
PPO-L, DDPG-H and even the unconstrained DDPG policy.
In Figure 3(c), we show constraint violations; the red increasing line shows cumulative liquidity demand in each [Figure] First 30 days Second 30 days, annualized values
Algorithms
Returns Sharpe Volatility MDD
Returns
Sharpe
Volatility
MDD
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
OLMAR
[Table]
[Table] constraints. NP policies nearly always outperform all other strategies. SP with α = 0 .
99 puts all funds in cash, hence MDD and volatility are 0, but returns are 0 as well.
[Figure]
period.
Although PPO-L, unlike CPO, was able to learn the constraints during training, both CPO and PPO-L are unable to come close to satisfying the liquidity constraints in testing (see the Supplementary Materials for constraint violations during training).
Only DDPG-H and NP satisfy the constraints in testing, but the DDPG-H method over-allocates to the liquid account, thereby reducing net returns.
ONLINE REPOSITIONING IN BIKE-SHARING
The bike repositioning problem is a form of online resource matching in an uncertain environment.
Uncoordinated movements of users in bike or electric vehicle sharing, along with demand uncertainty, results in the need to often reposition the resources [Ghosh et al., 2017, Schuijbroek et al., 2017, Ghosh et al., 2016, Ghosh and Varakantham, 2017].
We use an RL-based simulator from Bhatia et al.
[2019] built upon the dataset of Hubway bike sharing system in Boston, consisting of 95 base stations and 760 bikes.
The state at time t includes the current allocated bikes in each station j ∈ { 1 . . .
J } and ξ t,j is the random customer demand at station j .
The action is a vector, x t = { x t, 1 , ..., x t,J } , that represents the percent allocations of bikes across all stations while respecting the constraint set
[Equation]
= 1 , t = 1 . . .
T } , where ˇ C j and ˆ C j denote the minimum and maximum bounds on the number of allocated bikes at station j , and N denotes the total number of bikes present in the system. The [Figure] [Figure]
objective function is represented by:
[Equation]
where L ( x t,j , ξ ) represents the amount of unfulﬁlled demand and R ( x t,j , ξ ) is the number of bikes picked up or dropped off at station j at time t in action x , which incurs a repositioning cost.
We use two months of data to train the models.
We evaluate the learnt policies on 3 consecutive days during the morning peak period (6AM–12PM) withdecision epochs, each having a duration of 30 minutes.
Figure 4(a) shows mean and standard error in cumulative reward over 3 test days from NP with CVaR of α = 0 . 0 , 0 . 99 ,
CPO and PPO-L, and unconstrained DDPG.
After training
CPO and PPO-L methods for 1 million episodes, they fail to perform at par with NP.
NP using expected reward (i.e., α = 0 . 0 ) and with CVAR of α = 0 .
99 improves cumulative reward by 11.3% and 7.9% over unconstrained DDPG.
Figure 4(b) shows the cumulative constraint violation where capacity violation per station costs 1 unit.
Only NP variants satisfy the constraints, while both CPO and PPO-L fail to satisfy the capacity constraints during the test period.
Finally, we provide sensitivity analysis by varying (a) convex combination parameter, κ ; and (b) number of scenarios sampled from the demand distribution.
Figure 5(a) shows the mean and standard error in cumulative reward for different κ that decreases over iterations of NP.
Recall that at κ = 1 and 0 , NP reduces to DDPG and pure SP, respectively.
The best performance is achieved with κ = 1 i 2 , which is our default setting in the experiments.
Figure 5(b) shows the cumulative reward of NP in the last ﬁve time steps, varying the number of scenarios.
As expected, the NP performance improves with the number of scenarios, as the approximate distribution and hence the scenario tree formulation tends towards the true distribution. However, the improvement happens in a concave manner and beyond some number of scenarios, the performance gain hits a plateau, implying that computation requirements remain reasonable to achieve a near-optimal solution with NP.
We thus use 200 scenarios for both the pure SP and NP in the default settings of experiments.
RELATED WORK
The neural-progressive hedging algorithm combines ofﬂine policy search with an online model-based phase to ﬁne-tune the policy so as to satisfy constraints and risk measures such as
CVaR.
We categorize the existing relevant research into three threads: (a) Combining model-free and model-based methods for performance improvement; (b) Constrained and risk-sensitive RL methods; and (c) Improving sequential decisions through warm starting and imitation learning.
Ensemble of model-free and model-based methods:
Model-based methods are prized for sample efﬁciency, but, as noted by Feinberg et al.
[2018], high-capacity models are “prone to over-ﬁtting in the low-data regimes where they are most needed", implying that the combination of model-based and model-free methods will be important for good performance in complex settings.
They propose, as do Buckman et al.
[2018], to rollout the learned model for use in value estimation of a model-free RL, in the latter reference using an ensemble of such models to estimate variance.
Ghosh et al.
[2021] demonstrate the efﬁciency of combining the model-based and model-free RL methods in a complex air trafﬁc control domain.
Lu et al.
Amos et al.
[2018], Tamar et al.
[2017], Kahn et al.
[2017] combine (online) planning models with model-free RL to explore more sparingly the state and action spaces.
Mansard et al.
[2018] suggest a structure similar to ours for controlling dynamical systems using models to initialize a model predictive control formulation, as a warm-start.
Lu et al.
[2019] develop Adaptive Online Planning (AOP) with a continuous model-free RL algorithm, TF3 [Fujimoto et al., 2018].
The goal is similar to ours – leveraging the responsiveness of online planning with reactive off-policy learning to make better decisions.
The approach is however different from ours – AOP uses a model-based policy when uncertainty is high and a reactive model-free policy when habitual behavior should sufﬁce.
Constrained and risk-sensitive RL methods: Garcıa and
Fernández [2015] surveyed safe RL methods which they classify as either optimization-based or handling safety in the exploration process.
Pham et al.
[2018] suggest after each policy update to project the current iterate onto the feasible set of safety constraints; since they assume that the safety constraints may not be known in advance, they propose a method to learn the parameters of a linear polytope.
Yang et al.
[2019] extend CPO method to solve constrained
RL by optimizing the reward function using TRPO and then projecting the solution onto the feasible region deﬁned by safety constraints, similar to Pham et al.
Chow et al.
[2015, 2017] model risk-constrained MDPs with a CVaR objective or chance constraints, and solve it by relaxing the constraints and using a policy gradient algorithm.
However, similar to CPO and Lagrangian-relaxed PPO, the constraints are not enforced during execution and need not be satisﬁed.
Most “safe" RL methods use an initial infeasible, unconstrained policy and iteratively render it feasible and locally optimal, e.g., Berkenkamp et al.
[2017] deﬁne an expanding “region of attraction" to guide safe exploration to improve the policy, whilst remaining feasible.
Imitation learning and warm start: NP can be viewed through the lens of imitation learning.
Gu et al.
[2016] use synthetic model-based “imagination" rollouts in the early iterations of deep RL training, which can be considered as a model-based warm-start.
This is the opposite of our approach, we propose a longer-horizon deep RL to warm start the online stochastic program.
Aggravate [Ross and Bagnell, 2014] and Aggravated [Sun et al., 2017], building on the seminal DAgger [Ross et al., 2011], involve iteratively mixing the learning step of a policy with an expert policy, in that, at iteration n , π n = β n π ∗ + (1 − β n )ˆ π n , where β → 0 as n → ∞ .
This is similar to the update step of NP which uses a convex combination of the expert and the learner policies, with damping.
Cheng et al.
[2018] take this one step further by deﬁning a framework with a mirror descent gradient update that reduces to imitation learning-based RL, depending on the choice of the gradient estimator; they introduce SLOLS, where the gradient is a convex combination of a policy gradient and an expert gradient.
Plato [Kahn et al., 2017] is similar with the roles of the expert and learner reversed: the Plato expert replans at each step to avoid catastrophic failure in training, while the learner is a neural network.
Sun et al.
[2018] propose combining imitation learning and RL with the aim of faster learning and improving beyond a sub-optimal expert.
The advantages achieved by the NP method in inverting the roles of expert and learner are the ability of the SP to enforce hard constraints and incorporate risk measures, and doing so in an explainable manner.
The NP warm start serves as an external expert to guide the SP in the early iterations to encourage convergence to a better solution by reshaping the objective itself.
CONCLUSION
The neural-progressive hedging (NP) method starts from an ofﬂine, unconstrained RL policy and iteratively enforces constraints and risk requirements using model-based stochastic programming.
It is thus a type of external point method.
We demonstrate the efﬁcacy of NP method on two real-world applications, in ﬁnance and logistics.
The NP method signiﬁcantly outperforms both constrained and unconstrained RL whilst handling both resource constraints and risk measures.
An important beneﬁt of the framework is its ease of implementation: NP method can be implemented using existing deep RL algorithms and commercial off-the-shelf optimization packages, and provides added transparency and explainability on the constraint satisfaction of the policy.
One interesting direction for future work is the online re-formulation of the problem to take into account a time-varying distribution of ξ .
Similarly, upper bounding the loss from such a re-solving policy would be of great interest.