  Many reinforcement learning algorithms rely on value estimation. However, the
most widely used algorithms -- namely temporal difference algorithms -- can
diverge under both off-policy sampling and nonlinear function approximation.
Many algorithms have been developed for off-policy value estimation which are
sound under linear function approximation, based on the linear mean-squared
projected Bellman error (PBE). Extending these methods to the non-linear case
has been largely unsuccessful. Recently, several methods have been introduced
that approximate a different objective, called the mean-squared Bellman error
(BE), which naturally facilities nonlinear approximation. In this work, we
build on these insights and introduce a new generalized PBE, that extends the
linear PBE to the nonlinear setting. We show how this generalized objective
unifies previous work, including previous theory, and obtain new bounds for the
value error of the solutions of the generalized objective. We derive an
easy-to-use, but sound, algorithm to minimize the generalized objective which
is more stable across runs, is less sensitive to hyperparameters, and performs
favorably across four control domains with neural network function
approximation.
