This paper establishes rates of universal approximation for the shallow neural tangent kernel (NTK): network weights are only allowed microscopic changes from random initialization, which entails that activations are mostly unchanged, and the network is nearly equivalent to its linearization. Concretely, the paper has two main contributions: a generic scheme to approximate functions with the NTK by sampling from transport mappings between the initial weights and their desired values, and the construction of transport mappings via Fourier transforms. Regarding the first contribution, the proof scheme provides another perspective on how the NTK regime arises from rescaling: redundancy in the weights due to resampling allows individual weights to be scaled down. Regarding the second contribution, the most notable transport mapping asserts that roughly 1 /δ 10d nodes are sufficient to approximate continuous functions, where δ depends on the continuity properties of the target function. By contrast, nearly the same proof yields a bound of 1 /δ 2d for shallow ReLU networks; this gap suggests a tantalizing direction for future work, separating shallow ReLU networks and their linearization. As eq. (1.2) is merely affine in the parameters, it is not shocking that gradient descent can be analyzed. What is shocking is firstly that gradient descent on eq. (1.1) with small learning rate will track the behavior of eq. (1.2), and secondly the weights hardly change as a function of m, Contributions. This work provides rates of function approximation for the NTK as defined in eq. (1.2), moreover in the "NTK setting": the transported weights must be near initialization, meaning τ j −w j = O( 1 /ǫ √ m). In more detail: Continuous functions (cf. Theorem 1.5). The main theorem packages the primary tools here to say: the NTK can approximate continuous functions arbitrarily well, so long as the width is at least 1 /δ 10d , where δ depends on continuity properties of the target function; moreover, the transports satisfy τ j −w j = O( 1 /ǫ √ m), and the ReLU network (eq. (1.1)) and its linearization (eq. (1.2)) stay close. Re-using many parts of the proof, a nearly-optimal rate 1 /δ 2d is exhibited for ReLU networks in Theorem E.1; this gap between ReLU networks and their NTK poses a tantalizing gap for future work. Approximation via sampling of transport mappings. The first component of the proof of Theorem 1.5, detailed in Section 2, is a procedure which starts with an infinite width network, and describes how sampling introduces redundancy in the weights, and automatically leads to the desired microscopic transports τ j −w j = O( 1 /ǫ √ m). As detailed in Theorem 2.1, the error between the infinite width and sampled networks is O(ǫ + 1 / √ m). In this way, the analysis provides another perspective on the scaling behavior and small weight changes of the NTK. Construction of transport mappings. The second component of the proof of Theorem 1.5, detailed in Section 3, is to construct explicit transport mappings for various types of functions. As detailed in Lemma 3.3, approximating continuous functions proceeds by constructing an infinite width network not directly for the target function f , but instead its convolution f * G α with a Gaussian G α with tiny variance α 2 . Care is needed in order to obtain a rate of the form 1 /δ O(d) , rather than, say, 1 /δ O(d/δ) . The main constructions are based on Fourier transforms. Rounding out the organization of this paper: this introduction will state the main summarizing result and its intuition, and then close with related work; Section 4 will describe certain odds and ends for approximating continuous functions which were left out from the main tools in Section 2 and Section 3; Section 5 will conclude with open problems and related discussion. Proofs are sketched in the paper body, but details are deferred to the appendices.
REF considered the problem of universal approximation by shallow ReLU NTK models; in doing so, they introduced the idea of constructing both finite-and infinite-width approximations to a given target function by applying a transport map to the weights sampled from a Gaussian prior.
Neural tangent kernels, transportation mappings, and universal approximation
ArXiv
ArXiv
Mathematics Computer Science