  Recent empirical results show that deep networks can approximate solutions to
high dimensional PDEs, seemingly escaping the curse of dimensionality. However
many open questions remain regarding the theoretical basis for such
approximations, including the number of parameters required. In this paper, we
investigate the representational power of neural networks for approximating
solutions to linear elliptic PDEs with Dirichlet Boundary conditions. We prove
that when a PDE's coefficients are representable by small neural networks, the
parameters required to approximate its solution scale polynomially with the
input dimension $d$ and are proportional to the parameter counts of the
coefficient neural networks. Our proof is based on constructing a neural
network which simulates gradient descent in an appropriate Hilbert space which
converges to the solution of the PDE. Moreover, we bound the size of the neural
network needed to represent each iterate in terms of the neural network
representing the previous iterate, resulting in a final network whose
parameters depend polynomially on $d$ and does not depend on the volume of the
domain.
