This paper investigates the approximation property of the neural network with unbounded activation functions, such as the rectified linear unit (ReLU), which is new de-facto standard of deep learning. The ReLU network can be analyzed by the ridgelet transform with respect to Lizorkin distributions, which is introduced in this paper. By showing two reconstruction formulas by using the Fourier slice theorem and the Radon transform, it is shown that the neural network with unbounded activations still holds the universal approximation property. As an additional consequence, the ridgelet transform, or the backprojection filter in the Radon domain, is what the network will have learned after backpropagation. Subject to a constructive admissibility condition, the trained network can be obtained by just discretizing the ridgelet transform, without backpropagation. Numerical examples not only support the consistency of the admissibility condition but also imply that some nonadmissible cases result in low-pass filtering.
More recently, REF used ridgelet transform analysis to study the approximation properties of two-layer neural networks with unbounded activation functions, including the ReLU.
Neural Network with Unbounded Activation Functions is Universal Approximator
Applied and Computational Harmonic Analysis, 43(2):233-268, 2017
Mathematics Computer Science