  The Neural Tangent Kernel (NTK) has recently attracted intense study, as it
describes the evolution of an over-parameterized Neural Network (NN) trained by
gradient descent. However, it is now well-known that gradient descent is not
always a good optimizer for NNs, which can partially explain the unsatisfactory
practical performance of the NTK regression estimator. In this paper, we
introduce the Weighted Neural Tangent Kernel (WNTK), a generalized and improved
tool, which can capture an over-parameterized NN's training dynamics under
different optimizers. Theoretically, in the infinite-width limit, we prove: i)
the stability of the WNTK at initialization and during training, and ii) the
equivalence between the WNTK regression estimator and the corresponding NN
estimator with different learning rates on different parameters. With the
proposed weight update algorithm, both empirical and analytical WNTKs
outperform the corresponding NTKs in numerical experiments.
