  We present a numerical method for generating the state-feedback control
policy associated with general undiscounted, constant-setpoint,
infinite-horizon, nonlinear optimal control problems with continuous state
variables. The method is based on approximate dynamic programming, and is
closely related to approximate policy iteration. Existing methods typically
terminate based on the convergence of the control policy and either require a
discounted problem formulation or demand the cost function to lie in a specific
subclass of functions. The presented method extends on existing termination
criteria by requiring both the control policy and the resulting system state to
converge, allowing for use with undiscounted cost functions that are bounded
and continuous. This paper defines the numerical method, derives the relevant
underlying mathematical properties, and validates the numerical method with
representative examples. A MATLAB implementation with the shown examples is
freely available.
