We study the dynamics of temporal-difference learning with neural network-based value function approximation over a general state space, namely, Neural TD learning. Existing analysis of neural TD learningrelies on either infinite width-analysis or constraining the network parameters in a (random) compact set; as a result, an extra projection step is required at each iteration. This paper establishes a new convergence analysis of neural TD learningwithout any projection . We show that the projection-free TD learning equipped with a two-layer ReLU network of any width exceeding poly(%DIFDELCMD < \nu%%% , 1/\epsilon) converges to the true value function with error \epsilon given poly(%DIFDELCMD < \nu%%% , 1/\epsilon) iterations or samples, where%DIFDELCMD < \nu %%% is an upper bound on the RKHS norm of the value function induced by the neural tangent kernel. Our sample complexity and overparameterization bounds are based on a drift analysis of the network parameters as a stopped random process in the lazy training regime .
In this paper, we study the dynamics of temporal difference learning with neural network-based value function approximation over a general state space, namely, Neural TD learning. %DIFDELCMD < \nu%%% %DIFDELCMD < \nu%%% %DIFDELCMD < \nu %%% We consider two practically used algorithms, projection-free and max-norm regularized Neural TD learning, and establish the first convergence bounds for these algorithms. An interesting observation from our results is that max-norm regularization can dramatically improve the performance of TD learning algorithms, both in terms of sample complexity and overparameterization. In particular, we prove that max-norm regularization appears to be more effective than \ell_2-regularization, again both in terms of sample complexity and overparameterization . The results in this work rely on a novel Lyapunov drift analysis of the network parameters as a stopped and controlled random process .
R
We
In this paper, we
R
temporal-difference
temporal difference
D
Existing analysis of neural TD learningrelies on either infinite width-analysis or constraining the network parameters in a (random) compact set; as a result, an extra projection step is required at each iteration. This paper establishes a new convergence analysis of neural TD learning
D
without any projection
D
. We show that the projection-free TD learning equipped with a two-layer ReLU network of any width exceeding poly(
D
, 1/\epsilon) converges to the true value function with error \epsilon given poly(
D
, 1/\epsilon) iterations or samples, where
R
is an upper bound on the RKHS norm of the value function induced by the neural tangent kernel. Our
We consider two practically used algorithms, projection-free and max-norm regularized Neural TD learning, and establish the first convergence bounds for these algorithms. An interesting observation from our results is that max-norm regularization can dramatically improve the performance of TD learning algorithms, both in terms of sample complexity and overparameterization. In particular, we prove that max-norm regularization appears to be more effective than \ell_2-regularization, again both in terms of
R
bounds are based on a
. The results in this work rely on a novel Lyapunov
R
random process in the lazy training regime
and controlled random process