M EAN -V ARIANCE E FFICIENT R EINFORCEMENT L EARNING BY E XPECTED Q UADRATIC U TILITY M AXIMIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
In reinforcement learning (RL) for sequential decision making under uncertainty, existing methods proposed for considering mean-variance (MV) trade-off suffer from computational difﬁculties in computation of the gradient of the variance term.
In this paper, we aim to obtain MV-efﬁcient policies that achieve Pareto efﬁciency regarding MV trade-off.
To achieve this purpose, we train an agent to maximize the expected quadratic utility function, in which the maximizer corresponds to the Pareto efﬁcient policy.
Our approach does not suffer from the computational difﬁculties because it does not include gradient estimation of the variance.
In experiments, we conﬁrm the effectiveness of our proposed methods.
I NTRODUCTION
Reinforcement learning (RL) trains intelligent agents to solve sequential decision-making problems (Puterman, 1994; Sutton & Barto, 1998).
While a typical objective is to maximize the expected cumulative reward, risk-aware RL has recently attracted much attention in real-world applications, such as ﬁnance and robotics (Geibel & Wysotzki, 2005; Garcıa & Fern ´ andez, 2015).
Various criteria have been proposed to capture a risk, such as Value at Risk (Chow & Ghavamzadeh, 2014;
Chow et al., 2017) and variance (Markowitz, 1952; Luenberger et al., 1997; Tamar et al., 2012; Prashanth & Ghavamzadeh, 2013).
Among them, we consider the mean-variance RL (MVRL) methods that attempt to train an agent while controlling the mean-variance (MV) trade-off (Tamar et al., 2012;
Prashanth & Ghavamzadeh, 2013; 2016; Xie et al., 2018;
Bisi et al., 2020; Zhang et al., 2021b).
Existing MVRL methods (Tamar et al., 2012; Prashanth & Ghavamzadeh, 2013; 2016; Xie et al., 2018), typically maximize the expected cumulative reward while keeping the variance of the cumulative reward at a certain level or, equivalently, minimize the variance while keeping the expected cumulative reward at a certain level.
These MVRL methods simultaneously estimate the expected reward or variance while training an agent and solve the constrained optimization problem relaxed by penalized methods.
These studies have reported that RL-based methods suffer from high computational difﬁculty owing to the double sampling issue (Section 3) when approximating the gradient of the variance term (Tamar et al., 2012; Prashanth & Ghavamzadeh, 2013; 2016).
To avoid this, Tamar et al.
(2012) and Prashanth & Ghavamzadeh (2013) proposed multi-time-scale stochastic optimization.
Further, Xie et al.
(2018) proposed a method based on the Legendre-Fenchel duality (Boyd & Vandenberghe, 2004).
Although these methods avoid the double sampling issue, as we experimentally report in Figure 2 of Section 6, there still remains the difﬁculty in training a policy.
To avoid these difﬁculties, this paper considers another approach for MVRL by focusing on obtaining a policy that is located on the Pareto efﬁcient frontier in the sense of MV trade-off; that is, we cannot increase the expected reward without increasing the variance and decrease the variance without decreasing the expected reward (Section 2.2).
To achieve this purpose, we propose training a RL agent by direct expected quadratic utility maximization (EQUMRL) with a policy gradient method (Williams, 1988; 1992; Sutton et al., 2000; Baxter & Bartlett, 2001).
We show that the maximizer of the objective in EQUMRL is Pareto efﬁcient in the sense of MV trade-off.
As an important property, EQUMRL does not suffer from the double-sampling problem because it does not include the variance estimation, which is a cause of the problem.
In conventional methods, we can also obtain an MV-efﬁcient policy when we succeed in solving the constraint problem.
However, as discussed in the work and shown in our experiments, such as Figure 2, those methods do not perform well due to the difﬁculty in computation.
Compared with them, EQUMRL is computationally friendly and experimentally returns more MV-efﬁcient policies.
In addition, the EQUMRL is well suited to ﬁnancial applications, such as portfolio management because in economic theory, using MV portfolio is justiﬁed as a maximizer of the expected quadratic utility function (Markowitz, 1952; Luenberger et al., 1997).
We introduce brief survey on expected quadratic utility maximization and ﬁnance in Appendix A.We list the following three advantages to the EQUMRL approach:
(i) our proposed EQUMRL is able to learn Pareto efﬁcient policies and has plenty of interpretations from various perspectives (Section 4.3); (ii) our proposed EQUMRL does not suffer from the double sampling issue by avoiding explicit approximation of the variance (Section 4); (iii) we experimentally show that our proposed EQUMRL returns more Pareto efﬁcient policies than existing methods (Section 6).
In the following sections, we ﬁrst formulate the problem setting in Sections 2–3.
Then, we propose the main algorithms in Section 4.
Finally, we investigate the empirical effectiveness in Section 6.
P ROBLEM SETTING
We consider a standard setting of RL, where an agent interacts with an unfamiliar, dynamic, and stochastic environment modeled by a Markov decision process (MDP) in discrete time.
We deﬁne an MDP as a tuple ( S , A , r, P, P 0 ) , where S is a set of states, A is a set of actions, r : S ×
A → R is a stochastic reward function with ﬁnite mean and variance, P : S × S ×
A → [0 , 1] is a transition kernel, and P 0 : S → [0 , 1] is an initial state distribution.
The initial state S 1 is sampled from P 0 .
Let π θ : A × S
→
[0 , 1] be a parameterized stochastic policy mapping states to actions, where θ is a tunable parameter, and we denote the parameter space by Θ .
At time step t , an agent chooses an action A t following a policy π θ ( · | S t ) .
We assume that the policy π θ is differentiable with respect to θ ; that is, ∂π θ ( s | a ) ∂θ exists.
Let us deﬁne the expected cumulative reward from time step t to u as E π θ [ G t : u ] , where G t : u = (cid:80)
u
− t i =
γ i r ( S t
+ i , A t + i ) , γ ∈ (0 , 1] is a discount factor and E π θ denotes the expectation operator over a policy π θ , and S 1 is generated from P 0 .
When γ = 1 , to ensure that the cumulative reward is well-deﬁned, we usually assume that all policies are proper (Bertsekas & Tsitsiklis, 1995); that is, for any policy π θ , an agent goes to a recurrent state S ∗ with probability 1 , and obtains 0 reward after passing the recurrent state S ∗ at a stopping time τ .
This ﬁnite horizon setting is called episodic MDPs (Puterman, 1994).
For brevity, we denote G t : u as G when there is no ambiguity.
In this paper, we consider the trade-off between the mean and variance of the reward.
Let us denote the variance of a random variable W under π θ by V π θ ( W ) .
Note that E π θ [ G ] and V π θ ( G ) are ﬁnite because r ( S t , A t ) has ﬁnite mean and variance.
2. T RAJECTORY VARIANCE PERSPECTIVE .
A direction of MVRL is to consider the trajectory mean E
π θ [ G t : u ] and trajectory variance V π θ ( G t : u ) :
[Equation]
A typical method for considering the MV trade-off is to train a policy under some constrains.
Tamar et al.
(2012), Prashanth & Ghavamzadeh (2013), and Xie et al.
(2018) formulated MVRL as
[Equation]
We call an algorithm based on this formulation a trajectory MV-controlled RL.
In the trajectory
MV-controlled RL
, the goal is to maximize the expected cumulative reward, while controlling the trajectory-variance at a certain level.
To be more precise, their actual constraint condition is V π θ ( G ) ≤ η .
However, if V π θ ( G ) =
η is feasible, the optimizer satisﬁes the equality in applications, where we need to consider MV trade-off, such as a ﬁnancial portfolio management.
Therefore, we only consider an equality constraint.
To solve this problem, Tamar et al.
Prashanth & Ghavamzadeh (2013), and Xie et al.
(2018) considered a penalized method deﬁned as max θ ∈ Θ
E π θ [ G ] − δg (cid:0) V π θ ( G ) − η (cid:1) , where δ > 0 is a constant and g : R → R is a penalty function, such as g ( x ) = x or g ( x ) = x
See Remark 1 for more details.
2. MV- EFFICIENT POLICY .
To consider the MV trade-off avoiding the computational difﬁculty, this paper aims to train a Pareto efﬁcient policy in the sense of the MV trade-off, where we cannot increase the mean (resp.
decrease the variance) without increasing the variance (resp.
decreasing the mean).
Following the existing literature mainly in economics, ﬁnance, and operations research (Luenberger et al., 1997), we deﬁne a trajectory MV-efﬁcient policy as a policy π θ such that there is no other policy with θ (cid:48) ∈ Θ , where E π θ [ G ] ≤ E π θ (cid:48)
[ G ] and V π θ [ G ] > V π θ (cid:48)
[ G ] , or V π θ [ G ] ≥ V π θ (cid:48)
[ G ] and E π
θ
[ G ] < E π θ (cid:48)
[ G ] .
Efﬁcient frontier is deﬁned as a set of the MV-efﬁcient policies.
From the deﬁnition of the MV-controlled policies, the trained policies belong to the efﬁcient frontier.
P OLICY GRADIENT AND DOUBLE SAMPLING ISSUE
In this paper, we consider training a policy by policy gradient methods.
In MVRL, we usually require the gradients of E π θ [ G ] and V π θ ( G ) = E (cid:2) ( G − E π
θ
[ G ]) 2 (cid:3) .
Tamar et al.
(2012) and Prashanth &
Ghavamzadeh (2013) show that the gradients of E π θ [ G ] and E π θ (cid:2) G 2 (cid:3) are given as ∇ θ E π θ
[ G ] =
E π θ (cid:34)
[Equation]
Because optimizing the policy π θ directly using the gradients is computationally intractable, we replace them with their unbiased estimators.
Suppose that there is a simulator generating a trajectory k with { ( S kt , A kt , r ( S kt , A kt )) }
τ k t
=1 , where τ k is the stopping time of the trajectory.
Then, we can construct unbiased estimators of E π θ [ G ] and E π θ (cid:2) G 2 (cid:3) as follows (Tamar et al., 2012):
[Equation]
where (cid:98) G k is a sample approximation of E π θ [ G ] at the episode k .
Besides, because ∇ θ (cid:0) E π θ [ G ] (cid:1) 2 = 2 E π θ [ G ] ∇ θ E π θ [ G ] , the gradient of the variance is given as ∇ θ V π θ ( G ) =
∇ θ (cid:16) E
[Equation]
[ G 2 ] − (cid:0)
[Equation]
G ] (cid:1) 2 (cid:17) =
[Equation]
[ G ] ∇ θ E π θ [ G ] . However, obtaining an unbiased estimator of ∇ θ (cid:0) E π θ [ G ] (cid:1) 2 = 2 E π θ [ G ] ∇ θ E π θ [ G ] is difﬁcult because it requires sampling from two different trajectories for approximating E
π θ [ G ] and ∇ θ E π θ [ G ] .
This issue is called double sampling issue and makes the optimization problem difﬁcult when we include the variance V π θ ( G ) into the objective function directly.
Tamar et al.
(2012) and Prashanth &
Ghavamzadeh (2013) reported this double sampling issue caused from the gradient estimation of the variance and proposed solutions based on multi-time-scale stochastic optimization.
Recall that they consider the penalized objective function max θ ∈ Θ
E π θ [ G ] − δg (cid:0) V π θ ( G ) − η ) to obtain MV-controlled policies, where the double sampling issue caused by the gradient of δg (cid:0) V π θ ( G ) − η ) .
EQUMRL WITH T RAJECTORY VARIANCE PERSPECTIVE
In social sciences, we often assume that an agent maximizes the expected quadratic utility for considering MV trade-off.
Based on this, we propose the EQUMRL for obtaining an MV-efﬁcient policy.
For a cumulative reward G , we deﬁne the quadratic utility function as u trajectory ( G ; α, β ) =
αG − 12 βG 2 , where α > 0 and β ≥ 0 .
In EQUMRL, we train a policy by maximizing the expected value of the quadratic utility function,
[Equation]
The EQUMRL is agnostic to the learning method, that is, we can implement it with various existing algorithms such as REINFORCE and actor-crtic.
4. EQUMRL AND MV EFFICIENCY
The expected quadratic utility function E π θ (cid:2) u trajectory ( G ; α, β ) (cid:3) is known to be Pareto efﬁcient in the sense of mean and variance when its optimal value satisﬁes E π θ [ G ] ≤ αβ .
In order to conﬁrm this, we can decompose the expected quadratic utility as
[Equation]
When a policy π ∈ Π is the maximizer of the expected quadratic utility, it is equivalent to an
MV-efﬁcient policy (Borch, 1969; Baron, 1977; Luenberger et al., 1997).
Following (Luenberger et al., 1997, p.237–239), we explain this as follows:
Among policies π θ with a ﬁxed mean E π θ [ G ] = µ , the policy with the lowest variance maximizes the expected quadratic utility function because E π θ (cid:2) u trajectory ( G ; α, β ) (cid:3) =
αµ − 12 βµ 2 − 12 β V
π
θ
( G ) is a monotonous decreasing function on V π θ ( G ) ; 2.
Among policies π θ with a ﬁxed variance V π θ ( G ) =
σ 2 and mean E π θ [ G ] ≤ αβ , the policy with the highest mean maximizes the expected quadratic utility function because E π θ (cid:2) u trajectory ( G ; α, β ) (cid:3) =
− 12 β (cid:16) E π θ [ G ] − αβ (cid:17)
+ α 2 2 β − 12 βσ 2 is a monotonous increasing function on E
π θ [ G ] ≤ αβ .
Based on the above property, we propose maximizing the expected quadratic utility function in RL; that is, training an agent to directly maximize the expected quadratic utility function for MV control instead of solving a constrained optimization.
We call the framework that makes the RL objective function an expected quadratic utility EQUMRL.
Unlike the expected cumulative reward maximization in the standard RL setting, at time t , it is desirable to include the past cumulative reward to the state S t because our objective function depends on it even given S t .
Let us consider the objective at time t with the inﬁnite horizon setting:
[Equation]
{+where C is a constant and recall that G 0: t − 1 = (cid:80) t − 1 i =
+}
γ i r ( S i , A 0 ) .Thus, for a better decision-making, we include the past cumulative reward into the state space.
4. I MPLEMENTATION OF EQUMRL
In this section, we introduce how to train a policy with the EQURL.
We deﬁned the objective function of the EQUMRL, and EQUMRL is an agnostic in learning method.
As examples, we show an implementation based on REINFORCE (Williams, 1992; Brockman et al., 2016) and Actor-Critic (AC) methods (Williams & Peng, 1991; Mnih et al., 2016).
We use unbiased estimators of the gradients deﬁned in (1).
REINFORCE-based trajectory EQUMRL.
For an episode k with the length n , the proposed algorithm replaces E π θ (cid:2) G (cid:3) and E π θ (cid:2) G 2 (cid:3) with the sample approximations (cid:80) nt =
1 γ t − 1 r ( S t , A t ) and (cid:0)(cid:80) nt =1 γ t
[Equation]
1 r ( S t , A t ) (cid:1) 2 , respectively (Tamar et al., 2012); that is, the unbiased gradients are given as (1).
Therefore, for a sample approximation (cid:98) G k of E π θ (cid:2) G 2 (cid:3) at the episode k , we optimize the policy with ascending the unbiased gradient
[Equation]
Here, we present three advantages of RL for EQUMRL.
The ﬁrst advantage concerns computation.
In EQUMRL, we do not suffer from the double sampling issue because the term ( E π θ [ G ]) 2 , which causes the problem, is absent, and we must only estimate the gradients of E π θ [ G ] and E π θ (cid:2) G 2 (cid:3) .
The second advantage is that it provides a variety of interpretations, as listed in Section 4.3.
The third advantage is the ease of theoretical analysis.
For example, referring to the results of Bertsekas &
Tsitsiklis (1996), we derive the following result on the convergence of the gradient, which can be applied to simple policy gradient algorithms, not only our proposed REINFORCE-based algorithm.
Theorem 1.
Consider an update rule such that θ i +1 ←
θ i + η i (cid:98) ∇ E π θi (cid:2) u trajectory {+( G ; α, β ) (cid:3) , where the learning rates η i are non-negative and satisfy (cid:80) ∞ i =
 η i = ∞ and (cid:80) ∞ i =
+}
η 2 i < ∞ .
Suppose that (a) episode always ﬁnishes in ﬁnite horizon n ; (b) the policy π θ has always bounded ﬁrst and second partial derivatives.
Then, lim i →∞ ∇ θ E π θi [ αG − 12 βG 2 ] = 0 almost surely.
It is expected that non-asymptotic results can be derived by restricting the policy class and the optimization algorithm as Agarwal et al.
(2020) and Zhang et al.
(2021a)
, but this is not the scope of this paper, which aims to provide a general framework.
AC-based trajectory EQUMRL.
Another implementation of the EQUMRL is to apply the AC algorithm
(Williams & Peng, 1991; Mnih et al., 2016).
For an episode k with the length n , following Prashanth & Ghavamzadeh (2013; 2016), we train the policy by a gradient deﬁned as [Equation] where
[Equation]
2 γ n
[Equation]
G kt :
[Equation]
+ n
[Equation]
M
[Equation]
ˆ
[Equation]
1) k
( S kt + n ) + γ 2 n M (2) ˆ ω
(2) k
( S kt + n ) , and M (1) ˆ ω (1) k ( S kt ) and M (2) ˆ ω (2) k ( S kt ) are models of E [ G t +1: ∞ ] and E [ G 2 t +1: ∞ ] with parameters ˆ ω (1) k and ˆ ω (2) k .
For more details, see Prashanth & Ghavamzadeh (2013; 2016).
Remark 1 (Existing approaches) .
For the double sampling issue, Tamar et al.
(2012) and Prashanth & Ghavamzadeh (2013; 2016) proposed multi-time-scale stochastic optimization.
Their approaches are known to be sensitive to the choice of step-size schedules, which are not easy to control (Xie et al., 2018).
Xie et al.
(2018) proposed using the Legendre-Fenchel dual transformation with coordinate descent algorithm.
First, based on Lagrangian relaxation, Xie et al.
(2018) set an objective function as max θ ∈ Θ
E π θ [ G ] − δ
( V π θ ( G ) − η ) .
Then, Xie et al.
(2018) transformed the objective function as max θ ∈ Θ ,y ∈ R 2 y (cid:0)
E π
θ [ G ] + 12 δ (cid:1) − y 2 − E π θ (cid:2) G 2 (cid:3) and trained an agent by solving the optimization problem via a coordinate descent algorithm.
However, this approach does not reﬂect the constraint η because the constraint condition η vanishes from the objective function.
This problem is caused by their objective function based on the penalty function g ( x ) =
x : E π θ [ G ] − δ ( V π θ ( G ) − η ) , where the ﬁrst derivative does not include η .
To avoid this problem, we need an iterative algorithm to decide an optimal δ or change g ( x ) from x
but it is not obvious how to incorporate them into the approach.
Remark 2 (Difference from existing MV approaches) .
Readers may assert that EQUMRL simply omits ( E π θ [ G ]) 2 from existing MV-controlled RL methods, which usually includes the explicit variance term in the objective function, and is the essentially the same.
However, there are signiﬁcant differences; one of the main ﬁndings of this paper is our formulation of a simpler RL problem to obtain an MV-efﬁcient policy.
Existing MV-controlled RL methods suffer from computational difﬁculties caused by the double sampling issue.
However, we can obtain MV-efﬁcient policy without going through the difﬁcult problem.
In addition, EQUMRL shows better performance in experiments even from the viewpoint of the constrained problem because it is difﬁcult to choose parameters to avoid the double sampling issue in existing approaches.
Thus, the EQUMRL has advantage in avoiding solving more difﬁcult constrained problems for considering MV trade-off.
4. I NTERPRETATIONS OF EQUMRL WITH G RADEINT E STIMATION
We can interpret EQUMRL as an approach for (i) a targeting optimization problem to achieve an expected cumulative reward ζ , (ii) an expected cumulative reward maximization with regularization, and (iii) expected utility maximization via Taylor approximation.
First, we can also interpret EQUMRL as mean squared error (MSE) minimization between a cumulative reward R and a target return ζ ; that is,
[Equation]
We can decompose the MSE into the bias and variance as
[Equation]
Thus, the MSE minimization (4) is equivalent to EQUMRL (3), where ζ = αβ .
The above equation provides an important implication for the setting of ζ .
If we know the reward is shifted by x , we only have to adjust ζ to ζ + x .
This is because ζ only affects the bias term in the above equation.
The equation also provides another insight.
If our assumption max π θ E π θ [ G ] ≤ ζ is violated, E π θ [ G ] will not be maximized with a ﬁxed variance and will be biased towards ζ ; that is, EQUMRL cannot ﬁnd the MV-efﬁcient policies.
In applications, we can conﬁrm whether the optimization works by checking whether average value of the empirically realized cumulative rewards is less than ζ .
Second, we can regard the quadratic utility function as an expected cumulative reward maximization with a regularization term deﬁned as E (cid:2) R 2 (cid:3) ; that is, minimization of the risk R ( π θ ) :
[Equation]
where ψ > 0 is a regulation parameter and ψ = β 2 α = 12 ζ .
As ψ → 0 ( ζ → ∞ ), R ( θ ) → − E π θ [ G ] .
Third, the quadratic utility function is the quadratic Taylor approximation of a smooth utility function u ( G ) because for G 0 ∈ R , we can expand it as u ( G )
≈ u ( G 0 )
+ U (cid:48) ( G 0 )
( G − G 0 )
+ U (cid:48)(cid:48) ( G 0 )
( G −
G 0 ) 2 + · · · ; that is, quadratic utility is an approximation of various risk-averse utility functions.
This property also supports the use of the quadratic utility function in practice (Kroll et al., 1984).
It also should be noted that the EQUMRL is closely related to the ﬁelds of economics and ﬁnance, where the ultimate goal is to maximize the utility of an agent, which is also referred to as an investor .
The quadratic utility function is a standard risk-averse utility function often assumed in ﬁnancial theory Luenberger et al.
(1997) to justify an MV portfolio; that is, an MV portfolio maximizes an investor’s utility function if the utility function is quadratic (see Appendix A).
Therefore, our approach can be interpreted as a method that directly achieves the original goal.
Remark 3 (Speciﬁcation of utility function (hyper-parameter selection)) .
Next, we discuss how to decide the parameters α and β , which are equivalent to ζ , ξ , and ψ .
The meanings are equivalent to constrained conditions of MVRL; that is, we predetermine these hyperparameters depending on our attitude toward risk.
For instance, we propose the following three directions for the parameter choice.
First, we can determine β 2 α = ψ based on economic theory or market research (Ziemba et al., 1974; Kallberg et al., 1983)
(Appendix A).
Luenberger et al.
(1997) proposed some questionnaires to investors for the speciﬁcation.
Second, we set ζ = 12 ψ as the targeted reward that investors aim to achieve.
Third, through cross-validation, we can optimize the regularization parameter ψ to maximize some criteria, such as the Sharpe ratio (Sharpe, 1966).
However, we note that in time-series related tasks, we cannot use standard cross-validation owing to dependency.
Therefore, in our experiments with a real-world ﬁnancial dataset, we show various results under different parameters.
Remark 4 (From MV-efﬁcient RL to MV-controlled RL) .
We can also apply the MV-efﬁcient RL method as the MV-controlled RL method.
The parameters of the expected quadratic utility function correspond to the variance that we want to achieve.
For example, the larger ζ is in (4), the larger the variance will be.
Although we do not know the explicit correspondence between the parameter of the expected quadratic utility function and the variance, it is possible to control the variance by choosing an appropriate policy from among those learned under several parameters.
In Figure 2 in
Section 6, we show the mean and variance of several policies trained with several parameters.
These results are measured with test data, but by outputting multiple candidates with the training data (ex. cross-validation), we can choose a policy with the desired variance.
EQUMRL UNDER PER - STEP VARIANCE PERSPECTIVE
Another direction of MVRL is to consider the trade-off between the per-step mean E π θ
[ r ( S t , A t )] and per-step variance V π θ ( r ( S t , A t )) , E π θ [ r ( S t , A t )] trade-off ⇐⇒ V π θ ( r ( S t , A t )) .
As well as the trajectory perspective, Bisi et al.
(2020) and Zhang et al.
(2021b)
proposed training a policy to maximize the penalized objective function, max θ ∈ Θ (cid:80) Tt = 1 γ t − 1 (cid:16) E π θ [ r ( S t , A t )] − λ V π
θ
( r ( S t , A t )) (cid:17) .
We call an algorithm with this formulation a per-step MV-controlled RL.
See Appendix B for more details.
We deﬁne a per-step MV-efﬁcient policy as a policy π θ such that there is no other policy with θ (cid:48) ∈ Θ , where, for each t , E π θ [ r ( S t , A t )] ≤ E π θ (cid:48)
[ r ( S t , A t )] and V π θ [ r ( S t , A t )] > V π θ (cid:48)
[ r ( S t , A t )] , or V π θ [ r ( S t , A t )] ≥ V π θ (cid:48)
[ r ( S t , A t )] and E π θ [ r ( S t , A t )] < E π θ (cid:48)
[ r ( S t , A t )] .
For constants α, β > 0 , we deﬁne the quadratic utility function for per-step reward setting as u per-step ( r ( S t , A t ); α, β ) =
αr ( S t , A t ) − βr 2 ( S t , A t ) .
For an ﬁnite horizon case with γ = 1 , we consider maximizing the expected cumulative quadratic utility function deﬁned as
[Equation]
=1 r 2 ( S t , A t ) (cid:105) . When applying REINFORCE-based algorithm to train an agent, the sample approximation of the gradient, for instance, is given as (cid:98) ∇ E π θ (cid:104)(cid:80) Tt =
1 u per-step ( r ( S kt , A kt ); α, β ) (cid:105) =
α (cid:80) Tt =1 r ( S kt , A kt ) ∇ θ log π θ ( S kt , A kt ) − β (cid:80) Tt =1 (cid:0) r ( S kt , A kt ) (cid:1) 2 ∇ θ log π θ ( S kt , A kt ) .
E XPERIMENTS
This section investigates the empirical performance of the proposed EQUMRL in trajectory variance setting using synthetic and real-world ﬁnancial datasets.
We train the policy by REINFORCE-based algorithm.
We conduct two experiments.
In the ﬁrst experiments, following Tamar et al.
(2012; 2014), and Xie et al.
, we conduct portfolio management experiments with synthetic datasets.
In the third experiment, we conduct a portfolio management experiment with a dataset of Fama & French (1992), a standard benchmark in ﬁnance.
In Appendix C, following Tamar et al.
(2012; 2014), and
Xie et al.
, we also conduct American-style option experiments with synthetic datasets.
We implemented algorithms following the Pytorch example 1 .
For algorithms using neural networks, we use a three layer perceptron, where the numbers of the units in two hidden layers are the same as that of the input node, and that of the output node is 2 .
We note that in all results, naively maximizing the reward or minimizing the variance do not ensure a better algorithm; we evaluate an algorithm based on how it controls the MV trade-off.
We denote the hyperparameter of the EQUMRL by ζ , which has the same meaning as ( α, β ) and ψ .
For all experiments, we adopt episodic MDPs; that is, γ = 1 .
6. P ORTFOLIO MANAGEMENT WITH A SYNTHETIC DATASET
Following Tamar et al.
(2012) and Xie et al.
(2018), we consider a portfolio composed of two asset types: a liquid asset with a ﬁxed interest rate r l , and a non-liquid asset with a time-dependent interest rate taking either r lownl or r highnl , and the transition follows a switching probability p switch .
An investor can sell the liquid asset at every time step t = 1 , 2 , . .
, T but the non-liquid asset can only be sold after the maturity W periods.
This means that when holding 1 liquid asset, we obtain r l per period; when holding 1 non-liquid asset at the t -th period, we obtain r lownl or r highnl at the t + W -th period.
Besides, the non-liquid asset has a risk of not being paid with a probability p risk ; that is, if the non-liquid asset defaulted during the W periods, we could not obtain any rewards by having the asset.
An investor can change the portfolio by investing a ﬁxed fraction w of the total capital M in the non-liquid asset at each time step.
A typical investment strategy is to construct a portfolio using both liquid and non-liquid assets for decreasing the variance.
Following Xie et al.
(2018), we set [Table] r l = 1 .
001 , r lownl = 1 . 1 , r highnl = 2 , p switch = 0 .
1 , p risk = 0 .
05 , W = 4 , w = 0 .
2 , and M = 1 .
As a performance metric, we use the average cumulative reward (CR) and its variance (Var) when investing for 50 periods.
We compare the EQUMRL with the REINFORCE, MV-controlled methods proposed by Tamar et al.
(2012) (Tamar), and Xie et al.
(2018) (Xie).
We denote the variance constraint of
Tamar et al.
(2012) as Var and Lagrange multiplier of Xie et al.
(2018) as λ .
For training Tamar, Xie, and EQUMRL, we set the Adam optimizer with learning rate 0 .
01 and weight decay parameter 0 .
For each algorithm, we report performances under various hyperparameters as much as possible.
First, we show CRs and Vars of the EQUMRL during the training process in Figure 1, where we conduct 100 trials on the test environment to compute CRs and Vars for each episode.
Here, we also show the performance of REINFORCE, which corresponds to the EQUMRL with ζ = α/β = ∞ .
As Figure 1 shows, the EQUMRL trains MV-controlled policies well depending on the parameter ζ .
Next, we compare the EQUMRL on the test environment with the REINFORCE,
Tamar, and Xie.
We conduct 100 , 000 trials on the test environment to compute CRs and Vars.
In Figure 2, we plot performances under several hyperparameters, where the horizontal axis denotes the Var, and the vertical axis denotes the CR.
Trained agents with a higher CR and lower Var are Pareto efﬁcient.
As the result shows, the EQUMRL returns more efﬁcient portfolios than the others in almost all cases.
We conjecture that this is because while the EQUMRL is an end-to-end optimization for obtaining an efﬁcient agent, the other methods consist of several steps for solving the constrained optimization, where those multiple steps can be sources of the suboptimal result.
We show CRs and Vars of some of their results in the upper table of Table 1.
The MSEs between ζ and CR are also shown in the lower table of Table 1, where we can conﬁrm that the EQUMRL succeeded in minimizing the MSEs.
6. P ORTFOLIO MANAGEMENT WITH A REAL - WORLD DATASET
We use standard benchmarks called Fama & French (FF) datasets 2 to ensure the reproducibility (Fama & French, 1992).
Among FF datasets, we use the FF25, FF48 and FF100 datasets, where the FF and FF 100 dataset includes 25 and 100 assets formed based on size and book-to-market ratio;
the
FF48 dataset contains 48 assets representing different industrial sectors.
We use all datasets covering monthly data from July 1980 to June 2020.
Consider an episodic MDP.
Let the action space be the set of m assets, and y a,t and w a,t be the return and the portfolio weight of an asset a at time t .
The reward (portfolio return) at time 1 ≤ t ≤ T is deﬁned as y t = (cid:80) ma
=1 y a,t w a,t − Λ (cid:80) ma
=1 | w a,t − w a,t − 1 | , where Λ = 0 .
001 is the penalty of the portfolio weight turnover(change of the action).
On the t -th period, the agent observes a state S t =
(( y a,t − 1 , ..., y a,t − 12 , w a,t − 1 {+) a ∈A , (cid:80) t − 1 s =
 y s ) , and decides a+} html
[Table]
portfolio weight ( w a,t ) a ∈A as w a,t = π ( a, s t ) .
Between periods t and t + 1 , the agent has an asset a with the ratio w a,t .See
{+Section 4.1 for the reason why we include sum t − 1 s =
+}
y s in the state.
Portfolio models.
We use the following portfolio models.
An equally-weighted portfolio (EW) weights the ﬁnancial assets equally (DeMiguel et al., 2009).
A mean-variance portfolio (MV) computes the optimal variance under a mean constraint (Markowitz, 1952).
To compute the mean vector and covariance matrix, we use the latest 10 years ( 120 months) data.
A Kelly growth optimal portfolio with ensemble learning (EGO) is proposed by Shen et al.
We set the number of resamples as m 1 = 50 , the size of each resample m 2 = 5 τ , the number of periods of return data τ = 60 , the number of resampled subsets m 3 = 50 , and the size of each subset m 4 = n 0 .
7 , where m is number of assets; that is, m = 25 in FF25, m = 48 in FF48 and m = 100 in FF100.
Portfolio blending via Thompson sampling (BLD) is proposed by Shen & Wang (2016).
We use the latest 10 years ( 120 months) data to compute for the sample covariance matrix and blending parameters.
Denote a method proposed by Tamar et al.
(2012) by Tamar, and choose the parameter var from { 15 , 30 , 60 } .
Denote a method proposed by Xie et al.
(2018) by Xie, and choose the parameter λ from { 10 , 100 , 1000 } .
Denote the REINFORCE-based trajectory EQUMRL by EQUMRL, and choose a parameter ζ from { 0 . 5 , 0 . 75 , 1 . 5 } .
Performance metrics.
We apply the following standard measures in ﬁnance for evaluation (Brandt, 2010).
The cumulative reward (CR), annualized risk as the standard deviation of return (RISK) and risk-adjusted return (R/R) are deﬁned as follows: CR = 1 /T (cid:80) Tt =1 y t , V = 1 /T (cid:80)
Tt =
( y t − CR) 2 , and R / R = √ 12 × CR / √ Var .
R/R is the most important measure for a portfolio strategy and is often referred to as the Sharpe Ratio (Sharpe, 1966).
We also evaluate the maximum draw-down (MaxDD), which is another widely used risk measure (Magdon-Ismail & Atiya, 2004).
In particular,
MaxDD is the largest drop from a peak deﬁned as MaxDD = min t ∈ [1 ,T ] (cid:16) 0 ,
[Equation] [Equation]
[Equation]
−(cid:17) , where W k is the cumulative return of the portfolio until time k ; that is, W t = (cid:81) tt (cid:48)
=1 (1 + y t (cid:48) ) .
Table 2 reports the performances of the portfolios.
In almost all cases, the EQUMRL achieves the highest R/R and the lowest MaxDD.
Therefore, we can conﬁrm that the EQUMRL has a high R/R, and avoids a large drawdown.
The real objective (minimizing variance with a penalty on return targeting) for Tamar, MVP, and EQUMRL is shown in Appendix D. Except for FF48’s MVP, the objective itself is smaller than that of EQUMRL.
Since the values of the objective are proportional to
R/R, we can empirically conﬁrm that the better optimization, the better performance.
C ONCLUSION
In this paper, we proposed EQUMRL for MV-efﬁcient RL.
Compared with the conventional MVRL methods, EQUMRL is computationally friendly.
The proposed EQUMRL also includes various interpretations, such as targeting optimization and regularization, which expands the scope of applications of the method.
We investigated the effectiveness of the EQUMRL compared with the standard RL and existing MVRL methods through experiments using synthetic and real-world datasets.
A P RELIMINARIES OF ECONOMIC AND FINANCIAL THEORY A. U TILITY THEORY
Utility theory is the foundation of the choice theory under uncertainty including economics and ﬁnancial theory.
A utility function u ( · ) measures agent’s relative preference for different levels of total wealth W .
According to Morgenstern & Von Neumann (1953), a rational agent makes an investment decision to maximize the expected utility of wealth among a set of competing feasible investment alternatives.
For simplicity, the following assumptions are often made for the utility function used in economics and ﬁnance.
First, the utility function is assumed to be at least twice continuous differentiable.
The ﬁrst derivative of the utility function (the marginal utility of wealth) is always positive, i.e., U (cid:48)
( W ) > 0 , because of the assumption of non-satiation.
The second assumption concerns risk attitude of the agents called “risk averse”.
When we assume that an agent is risk averse, the utility function is described as a curve that increases monotonically and is concave.
The most often used utility function of a risk-averse agent is the quadratic utility function as follows:
[Equation]
(5) where α ≥ 0 , β > 0 .
Taking the expected value of the quadratic utility function in (5) yields:
[Equation]
Substituting E [ W 2 ]
=
V [ W ] + E
[ W ] 2 into (5) gives
[Equation]
Equation (7) shows that expected quadratic utility can be described in terms of mean E
[ W ] and variance V [ W ] of wealth.
Therefore, the assumption of a quadratic utility function is crucial to the mean-variance analysis.
Remark 5 (Approximation by quadratic utility function) .
Readers may be interested in how the quadratic utility function approximates other risk averse utility functions.
Kroll et al.
(1984) empirically answered this question by comparing MV portfolio (maximizer of expected quadratic utility function) and maximizers of other utility functions.
In their study, maximizers of other utility functions also almost located in MV Pareto efﬁcient frontier; that is, expected quadratic utility function approximates other risk averse utility function well.
Remark 6 (Non-vNM utility functions) .
Unlike MV trade-off, utility functions maximized by some recently proposed new risk criteria, such as VaR and Prospect theory, do not belong to traditional vNM utility function.
A. M ARKOWITZ ’ S PORTFOLIO
Considering the mean-variance trade-off in a portfolio and economic activity is an essential task in economics as Tamar et al.
(2012) and Xie et al.
(2018) pointed out.
The mean-variance trade-off is justiﬁed by assuming either quadratic utility function to the economic agent or multivariate normal distribution to the ﬁnancial asset returns (Borch, 1969; Baron, 1977; Luenberger et al., 1997).
This means that if either the agent follows the quadratic utility function or asset return follows the normal distribution, the agent’s expected utility function is maximized by maximizing the expected reward and minimizing the variance.
Therefore, the goal of Markowitz’s portfolio is not only to construct the portfolio itself but also to maximize the expected utility function of the agent.
See Figure 3.
Markowitz (1952) proposed the following steps for constructing a MV controlled portfolio (Also see
Markowitz (1959), page 288, and Luenberger et al. (1997)):
• Constructing portfolios minimizing the variance under several reward constraint; • Among the portfolios constructed in the ﬁrst step, the economic agent chooses a portfolio maximizing the utility function.
Conventional ﬁnancial methods often adopt this two-step approach because directly predicting the reward and variance to maximize the expected utility function is difﬁcult; therefore, ﬁrst gathering information based on analyses of an economist, then we construct the portfolio using the information and provide the set of the portfolios to an economic agent.
However, owing to the recent development of machine learning, we can directly represent the complicated economic dynamics using ﬂexible models, such as deep neural networks.
In addition, as Tamar et al.
(2012) and Xie et al.
(2018) reported, when constructing the mean-variance portfolio in RL, we suffer from the double sampling issue.
Therefore, this paper aims to achieve the original goal of the mean-variance approach; that is, the expected utility maximization.
Note that this idea is not restricted to ﬁnancial applications but can be applied to applications where the agent utility can be represented only by the mean and variance.
A. M ARKOWITZ ’ S PORTFOLIO AND CAPITAL ASSET PRICING MODEL
Markowitz’s portfolio is known as the mean-variance portfolio (Markowitz, 1952; Markowitz &
Todd, 2000).
Constructing the mean-variance portfolio is motivated by the agent’s expected utility maximization.
When the utility function is given as the quadratic utility function, or the ﬁnancial asset returns follow the multivariate normal distribution, a portfolio maximizing the agent’s expected utility function is given as a portfolio with minimum variance under a certain standard expected reward.
The Capital Asset Pricing Model (CAPM) theory is a concept which is closely related to Markowitz’s portfolio (Sharpe, 1964; Lintner, 1965; Mossin, 1966).
This theory theoretically explains the expected return of investors when the investor invests in a ﬁnancial asset; that is, it derives the optimal price of the ﬁnancial asset.
To derive this theory, as well as Markowitz’s portfolio, we assume the quadratic utility function to the investors or the multivariate normal distribution to the ﬁnancial assets.
Merton (1969) extended the static portfolio selection problem to a dynamic case.
Fishburn &
Burr Porter (1976) studied the sensitivity of the portfolio proportion when the safe and risky asset distributions change under the quadratic utility function.
Thus, there are various studies investigating relationship between the utility function and risk-averse optimization (Tobin, 1958; Kroll et al., 1984;
Bulmus¸ & ¨Ozekici, 2014; Bodnar et al., 2015a;b).
A. MV PORTFOLIO AND MVRL
Traditional portfolio theory have attempted to maximize the expected quadratic utility function by providing MV portfolios.
This is because MV portfolio is easier to interpret than EQUMRL, and we can obtain a solution by quadratic programming.
One of the main goals of MVRL methods is also to construct MV portfolio under a dynamic environment (Tamar et al., 2012).
However, we conjecture that there are three signiﬁcant differences between them.
First, unlike static problem,
MVRL suffers computational difﬁculties.
Second, while static problem solves quadratic programming given the expected reward and variance, MVRL methods simultaneously estimate these values and solve the constrained problem.
Third, while static MV portfolios gives us an exact solution of the constrained problem, MVRL often relaxes the constrained problem by penalized method, which causes approximation errors.
In particular, for the second and third points, the difference in how to handle the estimators of expected reward and variance is essential.
A. E MPIRICAL STUDIES ON THE UTILITY FUNCTIONS
The standard ﬁnancial theory is built on the assumption that the economic agent has the quadratic utility function.
For supporting this theory, there are several empirical studies to estimate the parameters of the quadratic utility function.
Markowitz & Todd (2000) discussed how the quadartic utility function approximates the other risk-averse utility functions.
Ziemba et al.
(1974) investigated the change of the portfolio proportion when the parameter of the quadratic utility function changes using the Canadian ﬁnancial dataset.
Recently, Bodnar et al.
(2018) investigate the risk parameter ( α and β in our formulation of the quadratic utility function) using the markets indexes in the world.
They found that the utility function parameter depends on the market data model.
A. C RITICISM
For the simple form of the quadratic utility function, the ﬁnancial models based on the utility are widely accepted in practice.
However, there is also criticism that the simple form cannot capture the real-world complicated utility function.
For instance, Kallberg et al.
(1983) criticized the use of the quadratic utility function and proposed using a utility function, including higher moments.
This study also provided empirical studies using U.S. ﬁnancial dataset for investigating the properties of the alternative utility functions.
However, to the best of our knowledge, ﬁnancial practitioners still prefer ﬁnancial models based on the quadratic utility function.
We consider this is because the simple form gains the interpretability of the ﬁnancial models.
A. E CONOMICS AND FINANCE
To mathematically describe an attitude toward risk, economics and ﬁnance developed expected utility theory, which assumes the Bernoulli utility function u ( R ) on an agent.
In the expected utility theory, an agent acts to maximize the von Neumann-Morgenstern (vNM) utility function
U ( F ( r ))
= (cid:82) u ( r ) dF ( r ) , where F ( r ) is a distribution of R (Mas-Colell et al., 1995).
We can relate the utility function form to agents with three different risk preferences: the utility function u ( R ) is concave for risk averse agents; linear for risk neutral agents; and convex risk seeking agents.
For instance, an agent with u ( R ) =
R corresponds to a risk neutral agent attempting to maximize their expected cumulative reward in a standard RL problem.
For more detailed explanation, see
Appendix A or standard textbooks of economics and ﬁnance, such as Mas-Colell et al.
(1995) and
Luenberger et al.
To make the Bernoulli utility function more meaningful, we assume that it is increasing function with regard to R ; that is, R ≤ αβ for all possible R .
Even without the assumption, for a given pair of ( α, β ) , a optimal policy maximizing the expected utility does not change; that is, the assumption is only related to interpretation of the quadratic utility function and does not affect the optimization.
On the other hand, the constraint condition is determined by an investor to maximize its expected utility (Luenberger et al., 1997).
Finally, in theories of economics and ﬁnance, investors can maximize their utility by choosing a portfolio from MV portfolios.
In addition, when R ≤ αβ for all possible value of R , a policy maximizing the expected utility function is also located on MV Pareto efﬁcient frontier.
B P ER - STEP VARIANCE PERSPECTIVE
Bisi et al.
(2020) deﬁnes the per-step reward random variable R , a discrete random variable taking the values in the image of r , by deﬁning its probability mass function as p ( R = x )
= (cid:80) s,a d π θ ( s, a )
[ r ( s, a ) = x ] , where 1 is the indicator function and d π θ is the normalized discounted state-action distribution, d π θ
=
(1 − γ ) (cid:80) ∞ t =0 γ t Pr µ 0 ,π θ ,p ( S t = s, A t = a ) .
Here, it is known that for γ < 1 , E π θ [ G ] = 11 − γ (cid:80) s,a d π θ ( s, a ) r ( s, a ) .
It follows that E π θ [ R ] =
(1 − γ )
J ( π θ ) .
Bisi et al.
(2020) showed that the per-step variance V π θ ( R ) ≤ V πθ ( r ( S t ,A t ))
(1 − γ ), which implies that the minimiza- tion of the per-step variance V π θ ( r ( S t , A t )) also minimizes trajectory-variance V π θ ( R ) .
Therefore, they train a policy π θ by maximizing J κ ( π ) =
E π θ [ r ( S t , A t )] − κ V π θ ( r ( S t , A t )) , where κ > 0 is a parameter of the penalty function.
Bisi et al.
(2020) reveals that J κ ( π ) =
E π θ [ R − κ ( R − E π θ [ R ]) 2 ] , which implies that optimizing J κ ( θ ) is equivalent to optimize the canonical risk-neutral objective of a new MDP with the same as the original MDP except that the new reward function r (cid:48) ( s, a ) =
r ( s, a ) − λ (cid:0) r
( s, a ) − (1 − γ )
J ( π ) (cid:1) 2 .
This reward function depends on the policy π , making the reward function nonstationary and conventional RL method unusable.
This problem is called policy-dependent-reward issue.
To solve this policy-dependent-reward issue, the methods of
Bisi et al.
(2020) and Zhang et al.
(2021b) are based on the trust region policy optimization (Schulman et al., 2015) and coordinate descent with Legendre-Fenchel duality (Xie et al., 2018), respectively.
For constants α, β > 0 , we deﬁne the quadratic utility function for per-step reward setting as u per-step ( r ( S t , A t ); α, β ) =
αr ( S t , A t ) − βr 2 ( S t , A t ) .
For an inﬁnite horizon case with γ < 1 , we consider maximizing the following expected cumulative quadratic utility function:
[Equation]
For an ﬁnite horizon case with γ = 1 , we consider maximizing the following expected cumulative quadratic utility function:
[Equation]
When applying REINFORCE-based algorithm to train an agent by maximizing the objective function, the sample approximation of the gradient, for instance, is given as
[Equation]
C A MERICAN - STYLE OPTION WITH A SYNTHETIC DATASET
An American-style option refers to a contract that we can execute an option right at any time before the maturity time τ ; that is, a buyer who bought a call option has a right to buy the asset with the strike price K call at any time; a buyer who bought a put option has a right to sell the with the strike price K put at any time.
In the setting of Tamar et al.
(2014) and Xie et al.
(2018), the buyer simultaneously buy call and put options, which have the strike price K call = 1 . 5 and K put = 1 . , respectively.
The maturity time is set as τ = 20 .
If the buyer executes the option at time t , the buyer obtains a reward r t = max(0 , K put − x t ) + max(0 , x t − W call ) , where x t is an asset price.
We set x 0 = 1 and deﬁne the stochastic process as follows:
x t = x t − 1 f u with probability 0 .
45 and x t = x t − 1 f d with probability 0 .
55 , where f u and f d .
These parameters follows Xie et al.
As well as Section 6.1, we compare the EQUMRL with policy gradient (EQUM) with the REINFORCE, Tamar, and Xie.
The other settings are also identical to Section 6.1.
We show performances under several hyperparameter in Figure 4 and CRs and Vars of some of their results in the upper table of Table 3.
We also denote MSE between ζ and achieved CR of in the lower table of Table 3.
From the table, we can ﬁnd that while EQUMRL minimize the MSE for lower ζ , the MSE of REINFORCE is smaller for higher ζ .
We consider this is because owing to the difﬁculty of MV control under this setting, naively maximizing the CR minimizes the MSE more than considering the MV trade-off for higher ζ .
[Table]
D D ETAILS OF EXPERIMENTS OF PORTFOLIO MANAGEMENT WITH A REAL - WORLD DATASET
Table 4 shows the performance of each portfolio model without the penalty of the turnover Λ = 0 .
We report both results with and without the penalty of the turnover in the following.
The average of real objective (minimizing variance with a penalty on return targeting) for Tamar,
MVP and EQUMRL from July 2000 to June 2020 is shown in Table 5.
We also divide the performance period into two for robustness checks.
Table 6 and 7 shows the ﬁrst-half results from July 2000 to June 2010 and the second-half results from July 2010 to June 2020.
In almost all cases, the EQUMRL achieves the highest R/R.
We plot results of Tamar, Xie, and EQUMRL with various parameters in Figure 5.
We choose the parameter of Tamar ( V ) from { 10 , 15 , 20 , 25 , 30 , 35 , 40 , 45 , 50 , 55 , 60 , 65 } , Xie ( λ ) from { 10 , 50 , 100 , 200 , 300 , 400 , 500 , 600 , 700 , 800 , 900 , 1000 } , and EQUMRL ( ζ ) from { 0 . 125 , 0 . 25 , 0 . 375 , 0 . 5 , 0 . 625 , 0 .
We only annotate the points of V = 10 , 25 , 40 , 55 , λ = 10 , 200 , 500 , 800 , and ζ = 0 . 125 , 0 . 5 , 0 .
Unlike Section 6.1, it is not easy to control the mean and variance owing to the difﬁculty of predicting the real-world ﬁnancial markets.
However, the EQUMRL tends to return more efﬁcient results than the other methods.
[Table] [Table]
[Table] [Table]
[Figure]