Neural tangent kernel
In the study of artificial neural networks (ANNs), the neural tangent kernel (NTK) is a kernel that describes the evolution of deep artificial neural networks during their training by gradient descent. It allows ANNs to be studied using theoretical tools from kernel methods. For most common neural network architectures, in the limit of large layer width the NTK becomes constant. This enables simple closed form statements to be made about neural network predictions, training dynamics, generalization, and loss surfaces. For example, it guarantees that wide enough ANNs converge to a global minimum when trained to minimize an  empirical loss. The NTK of large width networks is also related to several other large width limits of neural networks. The NTK was introduced in 2018 by Arthur Jacot, Franck Gabriel and Clément Hongler. It was implicit in contemporaneous work on overparameterization. Definition. Scalar output case. An ANN with scalar output consists of a family of functions formula_1 parametrized by a vector of parameters formula_2. The NTK is a kernel formula_3 defined byformula_4In the language of kernel methods, the NTK formula_5 is the kernel associated with the feature map formula_6. Vector output case. An ANN with vector output of size  formula_7 consists in a family of functions formula_8 parametrized by a vector of parameters formula_2. In this case, the NTK formula_10 is a matrix-valued kernel, with values in the space of formula_11 matrices, defined byformula_12 Derivation. When optimizing the parameters formula_2 of an ANN to minimize an empirical loss through gradient descent, the NTK governs the dynamics of the ANN output function formula_14 throughout the training. Scalar output case. For a dataset formula_15 with scalar labels formula_16 and a loss function formula_17, the associated empirical loss, defined on functions formula_18, is given byformula_19When the ANN formula_20 is trained to  fit the dataset (i.e. minimize formula_21) via continuous-time gradient descent, the parameters formula_22 evolve through the ordinary differential equation: During training the ANN output function follows an evolution differential equation given in terms of the NTK: This equation shows how the NTK drives the dynamics of formula_25 in the space of functions formula_26 during training. Vector output case. For a dataset formula_15 with vector labels formula_28 and a loss function formula_29, the corresponding empirical loss on functions formula_30 is defined byformula_19The training of formula_32 through continuous-time gradient descent yields the following evolution in function space driven by the  NTK:formula_33 Interpretation. The NTK formula_34 represents the influence of the loss gradient formula_35 with respect to example formula_36 on the evolution of ANN output formula_37 through a gradient descent step: in the scalar case, this readsformula_38In particular, each data point formula_39 influences the evolution of the output formula_37 for each formula_41 throughout the training, in a way that is captured by the NTK formula_34. Large-width limit. Recent theoretical and empirical work in deep learning has shown the performance of ANNs to strictly improve as their layer widths grow larger. For various ANN architectures, the NTK yields precise insight  into the training in this large-width regime. Wide fully-connected ANNs have a deterministic NTK, which remains constant throughout training. Consider an ANN with fully-connected layers formula_43 of widths formula_44, so that formula_45, where formula_46 is the composition of an affine transformation formula_47 with the pointwise application of a nonlinearity formula_48, where formula_49 parametrizes the maps formula_50. The parameters formula_2 are initialized randomly, in an independent, identically distributed way. As the widths grow, the NTK's scale is affected by the exact parametrization of the formula_47's and by the parameter initialization. This motivates the so-called NTK  parametrization formula_53. This parametrization ensures that if the parameters formula_2 are initialized as standard normal variables, the NTK has a finite nontrivial limit. In the large-width limit, the NTK converges to a deterministic (non-random) limit formula_55, which stays constant in time. The NTK formula_55 is explicitly given by formula_57, where formula_58 is determined by the set of recursive equations: where formula_60 denotes the kernel defined in terms of the Gaussian expectation: In this formula the kernels formula_62 are the ANN's so-called activation kernels. Wide fully connected networks are linear in their parameters throughout training. The NTK  describes the evolution of neural networks under gradient descent in function space. Dual to this perspective is an understanding of how neural networks evolve in parameter space, since the NTK is defined in terms of the gradient of the ANN's outputs with respect to its parameters. In the infinite width limit, the connection between these two perspectives becomes especially interesting. The NTK remaining constant throughout training at large widths co-occurs with the ANN being well described throughout training by its first order Taylor expansion around its parameters at initialization: Other architectures. The NTK can be studied for various  ANN architectures, in particular convolutional neural networks (CNNs), recurrent neural networks (RNNs) and transformers. In such settings, the large-width limit corresponds to letting the number of parameters grow, while keeping the number of layers fixed: for CNNs, this involves letting the number of channels grow. Applications. Convergence to a global minimum. For a convex loss functional formula_21 with a global minimum, if the NTK remains positive-definite during training, the loss of the ANN formula_65 converges to that minimum as formula_66. This positive-definiteness property has been shown in a number of cases, yielding the first proofs that large -width ANNs converge to global minima during training. Kernel methods. The NTK gives a rigorous connection between the inference performed by infinite-width ANNs and that performed by kernel methods: when the loss function is the least-squares loss, the inference performed by an ANN is in expectation equal to the kernel ridge regression (with zero ridge) with respect to the NTK formula_55. This suggests that the performance of large ANNs in the NTK parametrization can be replicated by kernel methods for suitably chosen kernels. Software libraries. Neural Tangents is a free and open-source Python library used for computing  and doing inference with the infinite width NTK and neural network Gaussian process (NNGP) corresponding to various common ANN architectures.