  Neural Networks (NNs) are the method of choice for building learning
algorithms. Their popularity stems from their empirical success on several
challenging learning problems. However, most scholars agree that a convincing
theoretical explanation for this success is still lacking.
  This article surveys the known approximation properties of the outputs of NNs
with the aim of uncovering the properties that are not present in the more
traditional methods of approximation used in numerical analysis. Comparisons
are made with traditional approximation methods from the viewpoint of rate
distortion. Another major component in the analysis of numerical approximation
is the computational time needed to construct the approximation and this in
turn is intimately connected with the stability of the approximation algorithm.
So the stability of numerical approximation using NNs is a large part of the
analysis put forward.
  The survey, for the most part, is concerned with NNs using the popular ReLU
activation function. In this case, the outputs of the NNs are piecewise linear
functions on rather complicated partitions of the domain of $f$ into cells that
are convex polytopes. When the architecture of the NN is fixed and the
parameters are allowed to vary, the set of output functions of the NN is a
parameterized nonlinear manifold. It is shown that this manifold has certain
space filling properties leading to an increased ability to approximate (better
rate distortion) but at the expense of numerical stability. The space filling
creates a challenge to the numerical method in finding best or good parameter
choices when trying to approximate.
