P OLICY G RADIENT WITH E XPECTED Q UADRATIC U TILITY M AXIMIZATION : A N EW M EAN -V ARIANCE A PPROACH IN R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
In real-world decision-making problems, risk management is critical.
Among various risk management approaches, the mean-variance criterion is one of the most widely used in practice.
In this paper, we suggest expected quadratic utility maximization (EQUM) as a new framework for policy gradient style reinforcement learning (RL) algorithms with mean-variance control.
The quadratic utility function is a common objective of risk management in ﬁnance and economics.
The proposed EQUM framework has several interpretations, such as rewardconstrained variance minimization and regularization , as well as agent utility maximization.
In addition, the computation of the EQUM framework is easier than that of existing mean-variance RL methods, which require double sampling .
In experiments, we demonstrate the effectiveness of the proposed framework in benchmark setting of RL and ﬁnancial data.
I NTRODUCTION
Reinforcement learning (RL) with Markov decision processes (MDPs) is one type of dynamic decision-making problem (Puterman, 1994; Bertsekas & Tsitsiklis, 1996; sut, 1998).
While the typical objective is the expected cumulative reward maximization, risk-aware decision-making has attracted great attention in real-world applications, such as ﬁnance and robotics (Geibel & Wysotzki, 2005; García & Fernández, 2015).
The notion of risk is related to the fact that even an optimal policy may perform poorly in some cases owing to the stochastic nature of the problem.
To capture the risk, various criteria have been proposed, such as Value at Risk (Luenberger, 1998; Chow &
Ghavamzadeh, 2014; Chow et al., 2017) and variance (Markowitz, 1952; Markowitz et al., 2000; Tamar et al., 2012; L.A. & Ghavamzadeh, 2013).
We focus on the mean-variance trade-off.
Typical mean-variance RL (MVRL) methods attempt to maximize the expected cumulative reward while maintaining the variance threshold (Tamar et al., 2012; L.A. & Ghavamzadeh, 2013; Prashanth & Ghavamzadeh, 2016; Xie et al., 2018;
Bisi et al., 2020; Zhang et al., 2020).
However, most existing MVRL methods suffer from high computational costs owing to the double sampling issue when approximating the gradient of the variance term (Tamar et al., 2012; L.A. & Ghavamzadeh, 2013; Prashanth & Ghavamzadeh, 2016).
To avoid the double sampling issue, Xie et al.
(2018) proposed a method based on the Legendre-Fenchel duality (Boyd & Vandenberghe, 2004).
Although the method does not suffer from the double sampling issue, we cannot apply a standard policy gradient method and must use a coordinate descent algorithm.
In addition, the method cannot control risk at a certain desirable level.
From an economics perspective, the difference in the RL objectives arises from the forms of utility functions.
When the objective of an agent is expected cumulative reward maximization, the utility function is risk-neutral ; when an agent attempts to control the risk based on an expected reward, the utility function is risk-averse (Mas-Colell et al., 1995).
In economics, there have been several risk-averse utility functions proposed.
The quadratic utility function is one such function and is frequently used in ﬁnancial economics (Luenberger, 1997).
Under the quadratic utility function, we can prove that the mean-variance optimization maximizes the utility of the agent.
In addition, the theory of the mean-variance optimization in ﬁnance is based on the quadratic utility maximization (Markowitz, 1952; Sharpe, 1964; Lintner, 1965; Mossin, 1966).
In this study, as one of the MVRL approaches, we consider the expected quadratic utility maximization (EQUM) base,d on the policy gradient (Williams, 1988; 1992; Sutton et al., 1999; Baxter & Bartlett, 2001).
EQUM has the following advantages: (i) low computational cost; (ii) numerous interpretations, and (iii) direct connections to real-world applications, such as ﬁnance.
In this study, as interpretations of
EQUM, we propose the minimization of the variance under the constraint of the expected cumulative reward, reward-targeting optimization, and regularization.
Thus, this study contributes to the context of risk-averse RL and MVRL.
In the following sections, we ﬁrst formulate the problem setting in
Section 2 and review the existing methods in Section 3.
Then, we propose the EQUM framework and main algorithm based on the simplest policy gradient (Brockman et al., 2016) in Section 4.
Finally, we investigate the effectiveness of EQUM in Section 5.
P ROBLEM S ETTING
We consider the standard RL framework, where a learning agent interacts with an unfamiliar, dynamic, and stochastic environment modeled by a Markov decision process (MDP) in discrete time.
We deﬁne the MDP as the tuple ( S , A , r, P, P 0 ) , where S is a set of states, A is a set of actions, r : S ×
A → R is a reward function, P : S × S ×
A → [0 , 1] is the transition kernel, and
P 0 : S → [0 , 1] is an initial state distribution.
The initial state S 1 is sampled from P 0 .
Let π θ : A × S
→
[0 , 1] be a parameterized stochastic policy mapping states to actions, where θ is the tunable parameter.
At time step t , an agent chooses an action A t according to a policy π θ ( · | S t ) .
We assume that the policy π θ is a differentiable function with respect to θ ; that is, ∂π θ ( a,s ) ∂θ exists.
There are several performance measures for a policy π θ .
One popular measure is the expected cumulative reward from time step t to u deﬁned as E π θ [ R t : u ] , where R t : u = (cid:80) ui =
γ i r ( S t
+ i , A t + i ) , γ ∈ [0 , 1] is a discount factor and E π θ denotes the expectation operator over a policy π θ , and Sis generated from P 0 .
When the discount factor γ = 1 , to ensure the cumulative reward well deﬁned, it is usually assumed that all policies are proper (Bertsekas & Tsitsiklis, 1996); that is, for any policy π θ , the agent goes to a recurrent state S ∗ with probability 1 .
After the agent passes the recurrent state S ∗ at a time τ , the rewards are always 0 .
Such a ﬁnite horizon case is called episodic MDPs (Puterman, 1994; Tamar et al., 2012).
For brevity, we denote R t : u as R when we do not distinguish ﬁnite and inﬁnite horizon cases.
Under these criteria, the agent may attempt to obtain a higher cumulative reward while taking higher risks.
In real-world applications, such as portfolio management in ﬁnance (Markowitz, 1952; Markowitz et al., 2000), such risky decision-making is not always desired, and we, therefore, consider the trade-off between the expected cumulative reward and the variance.
Thus, while the goal of the risk-neutral MDP problem is to ﬁnd the parameter θ that maximizes the total reward, we consider the mean-variance trade-off between the cumulative expected reward and the variance of the cumulative reward R in the risk-averse MVRL problem.
Note that even when the reward r is deterministic, the cumulative reward is a random variable owing to the stochastic policy, and there exists the mean-variance trade-off.
In addition, even if the optimal policy is deterministic, the proposed method empirically has the potential to improves the stability of the training from the observations in the experiments of Section 5.3.
E XISTING MVRL M ETHODS
In this section, we introduce existing studies of MVRL.
3. C ONSTRAINED T RAJECTORY -V ARIANCE P ROBLEM
Tamar et al.
(2012), L.A. & Ghavamzadeh (2013), and Xie et al.
(2018) formulated MVRL by a constrained optimization problem deﬁned as max θ ∈ Θ
E π θ [ R ] s . t .
Var π θ ( R ) ≤ η .
In their formulation, the goal is to maximize the expected cumulative reward with controlling the trajectoryvariance at a certain level.
To solve this problem, Tamar et al.
(2012), L.A. & Ghavamzadeh (2013), and Xie et al.
(2018) consider a penalized method deﬁned as max θ ∈ Θ
E π θ [ R ] − δg (cid:0) Var π θ ( R ) − η (cid:1) , where δ > 0 is a constant and g : R → R is a penalty function, such as g ( x ) = x or g ( x ) = x 2 .
3.1. D OUBLE S AMPLING I SSUE
Tamar et al.
(2012), L.A. & Ghavamzadeh (2013), and Xie et al.
(2018) report the double sampling issue in MVRL, which requires sampling from two different trajectories for estimating the policy gradient.
For instance, in an episodic MDP with the discount factor γ = 1 and the stopping time τ = min { t | S t = S ∗ } , the gradients of E π θ
[ R ] , E π θ (cid:2) R 2 (cid:3) , and (cid:0) E π θ
[ R ] (cid:1) 2 are given as ∇ θ E [Equation]
R [Equation] [Equation] [Equation]
2 (cid:80)
[Equation] log [Equation] ( S
[Equation]
[Equation] ) (cid:3) , and ∇ θ (cid:0) E π θ [ R ] (cid:1) 2 = 2 E π θ [ R ] ∇ θ E π θ
[ R ] (Tamar et al., 2012).
Besides, the gradient of the variance is given as E π θ (cid:2) R 2 (cid:80) τt =1 ∇ θ log π θ ( S t , A t ) (cid:3) − 2 E π θ [ R ] ∇ θ E π θ [ R ] .
Because optimizing the policy π θ using the gradients directly is computationally intractable, we replace them with their unbiased estimators.
Suppose that there is a simulator generating a trajectory k with { ( S kt , A kt , r ( S kt , A kt )) }
τ k t
=1 , where τ k is the stopping time of the trajectory.
Then, we can naively construct unbiased estimators of E π θ [ R ] and E π θ (cid:2) R 2 (cid:3) as (cid:98) ∇ θ E π θ [ R ] = (cid:98)
R k (cid:80) τ k t =1 ∇ θ log π θ ( S kt , A kt ) and (cid:98) ∇ θ E π θ (cid:2) R 2 (cid:3) = (cid:16) (cid:98) R k (cid:17) 2 (cid:80)
τ k t =1 ∇ θ log π θ ( S kt , A kt ) , where (cid:98) R k is a sample approximation of E π θ
[ R ] at the episode k .
However, obtaining an unbiased estimator of ∇ θ (cid:0) E π θ [ R ] (cid:1) 2 = 2 E π θ
[ R ] ∇ θ E π θ
[ R ] is difﬁcult because it requires sampling from two different trajectories for approximating E
π θ [ R ] and ∇ θ E π θ [ R ] , respectively.
This double sampling issue makes the optimization problem difﬁcult.
3.1. E XISTING S OLUTIONS TO D OUBLE S AMPLING I SSUE
For the double sampling issue, Tamar et al.
(2012), L.A. & Ghavamzadeh (2013), and Xie et al.
(2018) presented the following solutions.
Multi-time-scale stochastic optimization: The proposed methods of Tamar et al.
(2012) and L.A.
& Ghavamzadeh (2013) are based on stochastic approximation to ﬁnd an equilibrium point of an ordinary differential equation and saddle point of the objective, respectively.
Coordinate descent optimization: Xie et al.
(2018) proposed using the Legendre-Fenchel dual transformation with coordinate descent algorithm.
First, based on Lagrangian relaxation, Xie et al.
(2018) set an objective function as max θ ∈ Θ
E π θ [ R ] − δ (Var π θ ( R ) − η ) .
Then, Xie et al.
(2018) transformed the objective function as max θ ∈ Θ ,y ∈ R 2 y (cid:0)
E π θ [ R ] + 12 δ (cid:1) − y 2 − E π θ (cid:2)
R 2 (cid:3) and estimated a parameter by solving the optimization problem via a coordinate descent algorithm.
Weakness of existing approaches: The multi-time-scale approaches proposed by Tamar et al.
(2012) and L.A. & Ghavamzadeh (2013) are known to be sensitive to the choice of the step-size schedules, which are not easy to be controlled (Xie et al., 2018).
The method proposed by Xie et al.
(2018) does not reﬂect the constraint condition η as shown above; that is, in the objective function of Xie et al.
(2018), there exits penalty coefﬁcient δ , but does not exist the constraint condition η . Note that the problem of Xie et al. (2018) is owing to their objective function based on the penalty function g ( x ) = x : E π θ [ R ] − δ
(Var π θ ( R ) − η ) , in which the ﬁrst derivative does not include η .
In addition, when using quadratic function for g ( x ) = x 2 to consider η , we cannot remove E [ R 2 ] even with Legendre-Fenchel dual by Xie et al. (2018); that is, the method of Xie et al. (2018) also suffers the double sampling issue. 3. C ONSTRAINED P ER -S TEP V ARIANCE P ROBLEM
Bisi et al.
(2020) and Zhang et al.
(2020) proposed solving a constrained per-step variance problem for MVRL.
Bisi et al.
(2020) showed that the per-step variance Var π θ ( R ) ≤ Var π θ ( r ( S t ,A t )) (1 − γ ), which implies that the minimization of the per-step variance Var( r ( S t , A t )) also minimizes trajectory-variance Var π θ ( R ) .
Therefore, they train a policy π θ by maximizing E π θ [ r ( S t , A t )] − κ Var π θ
( r ( S t , A t )) , where κ > 0 is a parameter of the penalty function.
The methods of Bisi et al.
(2020) and Zhang et al.
(2020) are based on the trust region policy optimization (Schulman et
3. C ONSTRAINED C UMULATIVE E XPECTED R EWARD P ROBLEM
While existing MVRL studies mainly focus on a constrained trajectory-variance problem, a constrained cumulative expected reward optimization is also frequently used in practical situations, such as ﬁnance (Markowitz, 1952; Markowitz et al., 2000).
In the constrained cumulative expected reward problem, we solve the following problem:
[Equation]
Our proposed EQUM framework is based on this motivation; that is, variance minimization.
As shown in the following section, from a computational perspective, there is a critical difference between the constrained cumulative expected reward and trajectory-variance problems.
EQUM F RAMEWORK
In this paper, as a novel approach for MVRL, we propose a Expected Quadratic Utility maximization
RL (EQUM).
In economic model, by using two parameters α > 0 and β > 0 , the quadratic utility function for the cumulative return R is deﬁned as U ( R ; α, β ) =
αR − 1 2 βR 2 for α > 0 , β ≥(Luenberger, 1998).
The quadratic utility function captures the preference of a risk-averse agent over the cumulative return R .
Let us consider the expected quadratic utility function deﬁned as
[Equation]
In the EQUM framework, we train a policy by maximizing the expected quadratic utility function.
4. I NTERPRETATIONS
Here, we introduce four interpretations of EQUM.
We can interpret the EQUM as an approach for (i) an expected utility maximization, (ii) a targeting optimization problem to achieve an expected cumulative reward ζ , (iii) a constrained trajectory-reward problem with a quadratic penalized function, and (iv) an expected cumulative reward maximization with regularization.
First, we discuss the connection with respect to training an agent to achieve a predeﬁned return.
Let ζ be a target return that the algorithm aims to achieve.
Then, we consider the mean squared error (MSE) minimization between the expected deviation of the return and the target ζ :
[Equation]
We can decompose the MSE into the bias and variance as follows:
[Equation]
Thus, the minimization problem (3) trains the policy π θ to consider the trade-off between the bias (cid:0) ζ − E π θ [ R ] (cid:1) 2 and variance E π θ (cid:104)(cid:0) E π θ [ R ] − R
[Equation]
2 (cid:105) (Section 9.5 of Luenberger (1998)).
Moreover, we ﬁnd that the EQUM is equivalent to the reward-targeting optimization when ζ = αβ ; that is,
[Equation]
Second, the bias-variance trade-off heuristically provides a solution to a constraint optimization problem (1) with the constraint ξ = ζ = αβ by solving the following penalized problem:
[Equation]
Third, we can regard the quadratic utility function as an expected cumulative reward maximization with a regularization term deﬁned as E (cid:2) R 2 (cid:3) ; that is, minimization of the risk R ( π θ ) :
[Equation]
where ψ > 0 is a regulation parameter and ψ = β 2 α = 12 ζ .
As ζ → ∞ , R ( π θ ) → − E π θ [ R ] .
4. M ERITS OF THE EQUM F RAMEWORK
In this section, we present two advantages of the EQUM framework.
The ﬁrst advantage is in computation.
The EQUM framework is an MVRL method.
However, compared with existing MVRL methods, which involve the double sampling issue, the computation of the EQUM framework is much simpler because we transform the MVRL problem (4) into the optimization problem without the term ( E π θ [ R ]) 2 (see (2) and (5)).
The second advantage is that it provides a variety of interpretations.
Because the EQUM framework can interpret economic theory, it is applicable for modeling economic dynamics.
In addition, as one of the MVRL methods, it is suitable for various real-world applications, such as ﬁnance (Deng et al., 2016) and playing games.
By regarding the proposed
EQUM as a regularization framework, we can combine it with existing RL methods.
4. I MPLEMENTATIONS OF THE EQUM FRAMEWORK
Here, we discuss the implementations of the EQUM framework.
Simplest policy gradient with EQUM With the EQUM framework, we introduce a main algorithm with simplest policy gradient (SPG) algorithm (Brockman et al., 2016), which is also called the REINFORCE algorithm (Williams, 1992).
For an episode k with length n , the method replaces the expectations E π θ (cid:2) R (cid:3) and E π θ (cid:2) R 2 (cid:3) with the sample approximations (cid:80) nt
=
1 γ t − 1 r ( S t , A t ) and (cid:0)(cid:80) nt
=
1 γ t − 1 r ( S t , A t ) (cid:1) 2 , respectively (Brockman et al., 2016).
Then, the unbiased gradients are (cid:98) ∇ θ E π θ [ R ] = (cid:98) R k n (cid:88) t = ∇ θ log π θ ( S kt , A kt ) and (cid:98) ∇ θ E π θ (cid:2) R 2 (cid:3) = (cid:16) (cid:98) R k
(cid:17)n (cid:88) t = ∇ θ log π θ ( S kt , A kt ) .
Therefore, for a sample approximation (cid:98) R k of E π θ (cid:2) R 2 (cid:3) at the episode k , we optimize the policy with ascending the following gradient:
[Equation]
Actor-critic with EQUM: For another combination with the EQUM framework, we apply an actor-critic (AC) based algorithms, which is also refereed to as the advantage actor-critic (A2C) algorithm (Williams & Peng, 1991; Mnih et al., 2016).
Extending the AC algorithm, for an episode k with the length n , we train the policy by a gradient deﬁned as
[Equation]
, where (cid:101) R kt : n =
[-=
-][Equation]
R kt : n + γ n +1 M (1) ˆ ω (1) k ( S kn ) , (cid:101) R k, 2 t : n = (cid:16) (cid:98) R kt : n (cid:17) 2 + γ n +1 M (2) ˆ ω (2) k
( S kn ) , and M (1) ˆ ω (1) k
( S kt ) and M (2) ˆ ω (2) k
( S k t ) are value functions approximating E
[ R t : ∞ ] and E [ R 2 t : ∞ ] with parameters ˆ ω (1) k and ˆ ω (2) k , respectively.
We can also use a value function baseline as a variant of AC algorithm; that is, (cid:101) R kt : n = (cid:98) R kt : n , (cid:101) R k, 2 t : n = (cid:98) R k, 2 t : n .
Note that we can heuristically extend our proposed framework EQUM to accept the other RL algorithms by regarding the EQUM as a regularization.
Determining α and β : Next, we discuss the parameter tuning of α and β , which are equivalent to ζ , ξ , and ψ ).
As explained, the parameter ψ has several meanings based on the interpretations of the EQUM, such as the quadratic utility function, targeting optimization, and constrained optimization.
In addition, from the regularization perspective, we can adjust the parameter to maximize the expected cumulative reward in the validation data.
Thus, we propose the following four directions for the parameter tuning.
First, in economic applications, such as ﬁnance, we choose β 2 α = ψ based on the theoretical economic assumptions and economic empirical studies. For instance, Capital Asset
Pricing Model (CAPM), which is one of the most popular economic models, is also base on the quadratic utility function (Sharpe, 1964; Lintner, 1965; Mossin, 1966).
Second, we set ζ = 12 ψ as the targeted reward to achieve.
Third, we regard the parameter ψ as the constrained problem (1).
Fourth, through cross-validation, we optimize the regularization parameter ψ .
E XPERIMENTS
This section investigates the empirical performance of the proposed EQUM with synthetic and realworld ﬁnancial datasets.
The goal of these experiments is to construct a portfolio with a wellcontrolled mean and variance through algorithms.
In addition, we also show experimental results using CartPole and Atari games. Note that the per-step rewards of portfolio selection are stochastic variables and those of CartPole and Atari games are deterministic variables.
However, even when the reward is deterministic, the cumulative reward has randomness owing to the stochastic policy. 5. E XPERIMENTS WITH S YNTHETIC D ATASETS
Following Tamar et al. (2012) and Xie et al.
(2018), we artiﬁcially generate portfolio datasets. Let us consider a portfolio composed of two types of assets: a liquid asset, which has a ﬁxed interest rate r l = 1 . 001 , and a non-liquid asset, which has a time-dependent interest rate r nl ( t ) ∈ { 1 . 1 , 2 } . While we can sell the liquid asset at every time step t = 1 , 2 , . .
, T , we can sell the non-liquid asset only after a maturity period of N steps; that is, when having 1 liquid asset, we obtain 1 . 001 per period; when having 1 non-liquid asset at the t -th period, we obtain 1 . 1 or 2 at the t + N -th periods. In addition, the non-liquid asset has some risk of not being paid with a probability p risk ; that is, if the non-liquid asset defaulted during the N periods, we cannot obtain any proﬁts by having the asset. In this setup, a typical investment strategy is to construct a portfolio using both liquid and non-liquid assets to control the mean and variance.
In our model, the investor may change his portfolio by investing a ﬁxed fraction α = 0 . 2 to the non-liquid asset at each time step. As a performance metric of the portfolios, we focus on the mean and variance of the cumulative reward when having the cash 1 at the ﬁrst period and investing the cash for 50 periods following an algorithm. Here, we investigate the sensitivity of the EQUM against the parameter ψ = 2 /ζ and compare the
EQUM with the standard SPG and AC algorithms. We apply the SPG algorithm with the EQUM (Section 4.3) to the synthetic datasets.
Here, we do not use the AC algorithm because it is not straightforward to extend the inﬁnite horizon algorithm to a ﬁnite horizon case owing to the existence of the value function. For ζ = 2 /ψ , we use ζ = 1 , 2 , 3 , 4 , 5 , 6 , 10 , 20 .
For instance, from the targeting optimization perspective, EQUM with ψ = 0 .
1 is equal to minimize the MSE between the cumulative reward and ζ = 5 = 1 / (2 ∗ 0 . 1) .
We train the model with 500 episodes.
In Figure 1, we calculate the average reward and standard deviation at each episode of the training process by conducting 1 , 000 trials.
In Table 1, using the trained model for the test environment, we show the
[Table]
average reward (AR) and standard deviation (SD) by conducting 1 , 000 trials.
From Figure 1 and
Table 1, we can clearly conﬁrm that the EQUM framework realize the control of the mean-variance trade-off.
As shown in the results, we can reduce the variance by increasing ζ .
The MSE from ζ is shown in Appendix A.1.
5. E XPERIMENTS WITH R EAL -W ORLD D ATASET
We use well-known benchmarks called Fama & French (1992) (FF) datasets to ensure the reproducibility of the experiment.
We use the FF25, FF48 and FF100 datasets.
The FF25 and FFdataset includes 25 and 100 portfolios formed based on size and book-to-market ratio; the FF dataset contains 48 portfolios representing different industrial sectors. We formulate the problem by an episodic MDP.
We use all datasets covering monthly data from July 1980 to June 2020.
The state is past 12 months returns of each asset and the action is deﬁned as portfolio weight; that is, the number of actions is equal to that of assets.
The reward is obtained as the portfolio return.
Here, the portfolio return at time 1 ≤ t ≤ T is deﬁned as y t = (cid:80) mj
=1 y j,t w j,t − 1 , where y j,t is the return of j asset at time t , w j,t − 1 is the weight of j asset in the portfolio at time t − 1 , and m is the number of assets.
The length of the episode is 1 years ( 12 months).
For the stochastic policy, we adopt a three-layer feed-forward neural network with the ReLU activation function where the number of units in each respective layer is equal to the number of assets, 100 , 50 , and the number of actions.
We use the softmax function for the output layer.
Portfolio models:
In our experiment, we use the following portfolio models.
An equally-weighted portfolio (EW) weights the ﬁnancial assets equally (DeMiguel et al., 2007).
A mean-variance portfolio (MV) computes the optimal variance under a mean constraint (Markowitz, 1952).
For computing the mean vector and covariance matrix, we use the latest 10 years (120 months) data.
An Kelly growth optimal portfolio with ensemble learning (EGO) is proposed by Shen et al.
We set the number of resamples as m 1 = 50 , the size of each resample m 2 = 5 τ , the number of periods of return data τ = 60 , the number of resampled subsets m 3 = 50 , and the size of each subset m 4 = n 0 .
7 , where m is number of assets; that is, m = 25 in FF25, m = 48 in FF48 and m = 100 in
FF100.
A portfolio blending via Thompson sampling (BLD) is proposed by Shen & Wang (2016).
We use the latest 10 years ( 120 months) data to compute for the sample covariance matrix and blending parameters.
A policy gradients with variance related risk criteria (Tamar) is proposed by Tamar et al.
We set the target variance terms as 500.
A block coordinate ascent algorithm proposed by Xie et al.
(2018), which is referred to as Mean-Variance Portfolio (MVP).
Then, let us denote the SPG with the EQUM framework as EQUM.
The parameter ψ is chosen from 1 / 3 , 2 / 3 , and 1 .
For optimizing Tamar, MVP and EQUM, we set the Adam optimizer with learning rate as 0 .
01 and weight decay parameter as 0 . 1 .
To train the parameters of network, we set the number of episode as 10 .
Each portfolio is updated by sliding one-month-ahead.
Performance metrics: The following measures widely used in ﬁnance to evaluate portfolio strategies (Brandt, 2010) are chosen.
The cumulative return (CR), annualized risk as the standard deviation of return (RISK) and risk-adjusted return (R/R) are deﬁned as follows:
CR = (cid:80) Tt =1 y t , RISK =
(cid:113) 12 T − 1 (cid:80)
Tt =
( y t − CR /T ) 2 , and R / R = 12 T × CR / RISK .
R/R is the most important measure for a portfolio strategy.
We also evaluate the maximum draw-down (MaxDD), which is another widely used risk measure Magdon-Ismail & Atiya (2004) for the portfolio strategy.
In particular, MaxDD is the largest drop from a peak deﬁned as MaxDD = min t ∈ [1 ,T ] (cid:16) 0 ,
[Equation]
, where W k is the cumulative return of the portfolio until time k ; that is, W t = (cid:81) tt (cid:48)
=1 (1 + y t (cid:48) ) .
Table 2 reports the performances of the portfolios.
In almost all cases, the EQUM portfolio achieves the highest R/R and the lowest MaxDD.
Therefore, we can conﬁrm that the EQUM portfolio has a [Table] high R/R, and avoids a large drawdown.
The real objective (minimizing variance with a penalty on return targeting) for Tamar, MVP, and EQUM is shown in Appendix A.2. Except for FF48’s MVP, the objective itself is smaller than EQUM’s. Since the values of the objective is the same as the RR, we can empirically conﬁrm that the better optimization, the better performance. 5. E XPERIMENTS WITH C ART P OLE AND A TARI G AMES
We also conduct experiments using CartPole and Atari games, where the reward is given as deterministic, and the randomness of the cumulative reward depends only on the stochastic policy.
We investigate the sensitivity of ψ = 1 / (2 ζ ) with CartPole and the compared the performance of the
EQUM framework with that of Tamar et al.
(2012) and Xie et al.
The results are shown in Appendix A.3. In many experimental results, contrary to our expectations, we observed that the EQUM also improves the expected cumulative reward, not only the variance.
We hypothesize that this is because there is often a limit on the cumulative rewards achieved by the standard expected cumulative reward maximizing algorithms.
For instance, when the reward at each period is 1 , and the discount factor is 0 . 99 , the inﬁnite sum is 100 .
In such a case, instead of naive reward maximization, MSE minimization against the target reward 100 may result in a more stable performance empirically.
We also hypothesize that even if an optimal policy is deterministic, the EQUM can improve the stability of the training process by reducing the variance.
This observation is an open problem related to exploration and exploitation trade-off.
Therefore, unless we could increase the cumulative reward inﬁnitely, the proposed EQUM framework can stabilize the performance.
From this aspect, we can conﬁrm that the EQUM provides a regularization effect.
C ONCLUSION
In this paper, we proposed an EQUM framework as a variant of MVRL.
Compared with existing
MVRL methods, the EQUM framework is computationally friendly.
The proposed EQUM framework also includes various interpretations, such as targeting optimization and regularization and is suitable for many real-world applications, such as ﬁnance and playing games.
We investigated the effectiveness of the EQUM framework compared with the standard RL and existing MVRL methods through the experiments.
In the results, the proposed method successfully controls the mean-variance trade-off. As an open problem, we also observed that even when an optimal policy is deterministic, the proposed method improves performance.
We hypothesize that the proposed method affects the stabilization of the training process.
In this section, we show the additional experiments and describe the details of the experiments.
To implement SPG and AC algorithms, we mainly follow the Pytorch example 1 .
A. A DDITIONAL R ESULTS OF E XPERIMENTS USING THE S YNTHETIC D ATASET
We show the MSE between the realized reward and the target reward ζ in Table 3.
The results represent the performance of the target-reward optimization.
The best performing method is highlighted in bold.
A. D ETAILS OF E XPERIMENTS OF P ORTFOLIO O PTIMIZATION
The real objective (minimizing variance with a penalty on return targeting) for Tamar, MVP and
EQUM is shown in Table 4. Except for FF48’s MVP, the objective itself is smaller than EQUM’s. Since the values of objective is the same as the RR, we can empirically conﬁrm that the better optimization, the better performance. We also divide the performance period into two for robustness checks.
Table 5 shows the ﬁrst-half results from July 2000 to June 2010 and the second-half results from July 2010 to June 2020.
In almost all cases, the EQUM portfolio achieves the highest R/R.
A. D ETAILS OF E XPERIMENTS WITH C AR P OLE AND A TARI G AMES
In this section, we report the experimental performances of the proposed EQUM framework with well-known benchmarks.
We investigate how the behaviors of existing RL methods are changed by adding the additional E [ R 2 ] term.
We use a simple two-layer perceptron for modeling the policy following the Pytorch example (Paszke et al., 2019). For the SPG-based algorithms, we [Table]
A.3. S ENSITIVITY A NALYSIS ON ψ
First, we investigate the sensitivity of the EQUM framework to the parameter ψ using the CartPole.
We apply the SPG and AC methods (Section 4.3) with the EQUM framework to the Cartpole problem, which is a classic control problem.
We use ψ = 0 .
and compare the results with the standard SPG and AC methods.
For instance, from the targeting optimization perspective, EQUM with ψ = 0 .
001 is equal to minimize the MSE between the cumulative reward and ζ = 500 = 1 / (2 ∗ 0 . 001) .
We train the model by 300 episodes.
We calculate the average reward and standard deviation at each period by conducting 1 , 000 trials.
The results are shown in Figure 2.
In the SPG results, we can conﬁrm the mean-variance trade-off.
In contrast, in the
SPG with ψ = 0 .
001 and all AC results, the EQUM framework improves the expected cumulative reward.
These results are contrary to our expectations because MVRL methods typically decrease the expected cumulative return to decrease the variance.
We discuss this topic in more detail in
Section 5.3.
A.3. E XPERIMENTS OF A TARI G AMES
When playing games, we often consider risk-control while maintaining a certain level of reward.
Under this motivation, we benchmark the proposed EQUM framework on four Atari game tasks from the OpenAI gym (Brockman et al., 2016).
Among the games, we choose BeamRider, Seaquest, Qbert, and SpaceInvaders in which SPG and AC methods work well.
We use simpliﬁed environments in which the observations are the RAM of the Atari machine, consisting of only 128 bytes.
[Figure] [Table] We compare our methods with the standard SPG, and MVLR methods of Tamar et al. (2012) and
Xie et al.
We calculate the average reward (AR) at the last episode over 5 trials and the standard deviation (SD).
In Table 6, we show the results of the SPG algorithm with the standard setting and EQUM framework, where ψ is chosen from 0 .
03 , and 0 . 3 .
Note that when using ψ = 0 .
005 , it is equal to minimize the MSE between the expected cumulative reward and the target 100 .
In almost all cases, the EQUM framework shows the better AR than the standard methods.
As well as the previous sensitivity experiment, this result is contrary to our expectation.
Among the games, we choose simpliﬁed BeamRider, Seaquest, Qbert, and SpaceInvaders in which the SPG algorithm work well, and the observations are the RAM of the Atari machine, consisting of only 128 bytes.
We compare our methods with the standard SPG algorithms, and MVRL methods of Tamar et al.
(2012) and Xie et al.
We denote the SPG-based methods proposed by Tamar et al.
(2012) and Xie et al.
(2018) as Tamar and MVP, respectively.
We calculate the average reward (AR) at the last episode over 5 trials and the standard deviation (SD).
We choose the parameter of
Tamar from 200 and 2000 , which is denoted as b in Tamar et al.
We choose the parameter of MVP from 1 and 10 , which is denoted as λ in Tamar et al.
The parameters are denoted as param in Table 6.