  We derive bounds on the error, in high-order Sobolev norms, incurred in the
approximation of Sobolev-regular as well as analytic functions by neural
networks with the hyperbolic tangent activation function. These bounds provide
explicit estimates on the approximation error with respect to the size of the
neural networks. We show that tanh neural networks with only two hidden layers
suffice to approximate functions at comparable or better rates than much deeper
ReLU neural networks.
