  While deep reinforcement learning has achieved tremendous successes in
various applications, most existing works only focus on maximizing the expected
value of total return and thus ignore its inherent stochasticity. Such
stochasticity is also known as the aleatoric uncertainty and is closely related
to the notion of risk. In this work, we make the first attempt to study
risk-sensitive deep reinforcement learning under the average reward setting
with the variance risk criteria. In particular, we focus on a
variance-constrained policy optimization problem where the goal is to find a
policy that maximizes the expected value of the long-run average reward,
subject to a constraint that the long-run variance of the average reward is
upper bounded by a threshold. Utilizing Lagrangian and Fenchel dualities, we
transform the original problem into an unconstrained saddle-point policy
optimization problem, and propose an actor-critic algorithm that iteratively
and efficiently updates the policy, the Lagrange multiplier, and the Fenchel
dual variable. When both the value and policy functions are represented by
multi-layer overparameterized neural networks, we prove that our actor-critic
algorithm generates a sequence of policies that finds a globally optimal policy
at a sublinear rate.
