  To further understand the underlying mechanism of various reinforcement
learning (RL) algorithms and also to better use the optimization theory to make
further progress in RL, many researchers begin to revisit the linear-quadratic
regulator (LQR) problem, whose setting is simple and yet captures the
characteristics of RL. Inspired by this, this work is concerned with the
model-free design of stochastic LQR controller for linear systems subject to
Gaussian noises, from the perspective of both RL and primal-dual optimization.
From the RL perspective, we first develop a new model-free off-policy policy
iteration (MF-OPPI) algorithm, in which the sampled data is repeatedly used for
updating the policy to alleviate the data-hungry problem to some extent. We
then provide a rigorous analysis for algorithm convergence by showing that the
involved iterations are equivalent to the iterations in the classical policy
iteration (PI) algorithm. From the perspective of optimization, we first
reformulate the stochastic LQR problem at hand as a constrained non-convex
optimization problem, which is shown to have strong duality. Then, to solve
this non-convex optimization problem, we propose a model-based primal-dual
(MB-PD) algorithm based on the properties of the resulting Karush-Kuhn-Tucker
(KKT) conditions. We also give a model-free implementation for the MB-PD
algorithm by solving a transformed dual feasibility condition. More
importantly, we show that the dual and primal update steps in the MB-PD
algorithm can be interpreted as the policy evaluation and policy improvement
steps in the PI algorithm, respectively. Finally, we provide one simulation
example to show the performance of the proposed algorithms.
