We consider continuous-time Mean-variance (MV) portfolio optimization problem in the Reinforcement Learning (RL) setting . The problem falls into the entropy-regularized  relaxed stochastic control framework recently introduced in Wang et al. (2019). We derive the feedback exploration policy as the Gaussiandistribution , with time-decaying variance. Close connections between the entropy-regularized MV and the classical MV are also discussed , including the solvability equivalence and the convergence as exploration decays . Finally, we prove a policy improvement theorem (PIT) for the continuous-time MV problem under both entropy regularization and control relaxation. The PIT leads to an implementable RL algorithm for the continuous-time MV problem. Our algorithm outperforms an adaptive control based method that estimates the underlying parameters in real-time and a state-of-the-art RL method that uses deep neural networks for continuous control problems by a large margin in nearly all simulations.
We approach the continuous-time mean-variance (MV) portfolio selection with reinforcement learning (RL) . The problem is to achieve the best tradeoff between exploration and exploitation, and is formulated as an entropy-regularized , relaxed stochastic control problem. We prove that the optimal feedback policy for this problem must be Gaussian , with time-decaying variance. We then establish connections between the entropy-regularized MV and the classical MV , including the solvability equivalence and the convergence as exploration weighting parameter decays to zero . Finally, we prove a policy improvement theorem , based on which we devise an implementable RL algorithm . We find that our algorithm outperforms both an adaptive control based method and a deep neural networks based algorithm by a large margin in our simulations.
R
consider
approach the
R
Mean-variance
mean-variance
R
optimization problem in the Reinforcement Learning
selection with reinforcement learning
D
setting
R
falls into the
is to achieve the best tradeoff between exploration and exploitation, and is formulated as an
A
,
R
framework recently introduced in Wang et al. (2019). We derive the feedback exploration policy as the Gaussiandistribution
problem. We prove that the optimal feedback policy for this problem must be Gaussian
R
Close
We then establish
D
are also discussed
R
decays
weighting parameter decays to zero
R
(PIT) for the continuous-time MV problem under both entropy regularization and control relaxation. The PIT leads to
, based on which we devise
R
for the continuous-time MV problem. Our algorithm outperforms
. We find that our algorithm outperforms both
R
that estimates the underlying parameters in real-time and a state-of-the-art RL method that uses
and a
R
for continuous control problems
based algorithm
R
nearly all
our