  We provide an entropy bound for the spaces of neural networks with piecewise
linear activation functions, such as the ReLU and the absolute value functions.
This bound generalizes the known entropy bound for the space of linear
functions on $\mathbb{R}^d$ and it depends on the value at the point
$(1,1,...,1)$ of the networks obtained by taking the absolute values of all
parameters of original networks. Keeping this value together with the depth,
width and the parameters of the networks to have logarithmic dependence on
$1/\varepsilon$, we $\varepsilon$-approximate functions that are analytic on
certain regions of $\mathbb{C}^d$. As a statistical application we derive an
oracle inequality for the expected error of the considered penalized deep
neural network estimators.
