Abstract-In this paper, we present an empirical study of iterative least squares minimization of the Hamilton-Jacobi-Bellman (HJB) residual with a neural network (NN) approximation of the value function. Although the nonlinearities in the optimal control problem and NN approximator preclude theoretical guarantees and raise concerns of numerical instabilities, we present two simple methods for promoting convergence, the effectiveness of which is presented in a series of experiments. The first method involves the gradual increase of the horizon time scale, with a corresponding gradual increase in value function complexity. The second method involves the assumption of stochastic dynamics which introduces a regularizing second derivative term to the HJB equation. A gradual reduction of this term provides further stabilization of the convergence. We demonstrate the solution of several problems, including the 4-D inverted-pendulum system with bounded control. Our approach requires no initial stabilizing policy or any restrictive assumptions on the plant or cost function, only knowledge of the plant dynamics. In the Appendix, we provide the equations for first-and second-order differential backpropagation. Index Terms-Differential neural networks (NNs), dynamic programming, feedforward neural networks, Hamilton-Jacoby-Bellman (HJB) equation, optimal control, viscosity solution.
Most similar to our approach, REF used neural networks of the Pineda architecture to learn the average cost solution of the HJB.
Least Squares Solutions of the HJB Equation With Neural Network Value-Function Approximators
IEEE Transactions on Neural Networks
IEEE Transactions on Neural Networks
Mathematics Medicine Computer Science