  In this paper, we study the generalization performance of min $\ell_2$-norm
overfitting solutions for the neural tangent kernel (NTK) model of a two-layer
neural network. We show that, depending on the ground-truth function, the test
error of overfitted NTK models exhibits characteristics that are different from
the "double-descent" of other overparameterized linear models with simple
Fourier or Gaussian features. Specifically, for a class of learnable functions,
we provide a new upper bound of the generalization error that approaches a
small limiting value, even when the number of neurons $p$ approaches infinity.
This limiting value further decreases with the number of training samples $n$.
For functions outside of this class, we provide a lower bound on the
generalization error that does not diminish to zero even when $n$ and $p$ are
both large.
