  Approximating the numerical solutions of Partial Differential Equations
(PDEs) using neural networks is a promising application of deep learning. The
smooth architecture of a fully connected neural network is appropriate for
finding the solutions of PDEs; the corresponding loss function can also be
intuitively designed and guarantees the convergence for various kinds of PDEs.
However, the rate of convergence has been considered as a weakness of this
approach. This paper introduces a novel loss function for the training of
neural networks to find the solutions of PDEs, making the training
substantially efficient. Inspired by the recent studies that incorporate
derivative information for the training of neural networks, we develop a loss
function that guides a neural network to reduce the error in the corresponding
Sobolev space. Surprisingly, a simple modification of the loss function can
make the training process similar to Sobolev Training although solving PDEs
with neural networks is not a fully supervised learning task. We provide
several theoretical justifications for such an approach for the viscous Burgers
equation and the kinetic Fokker--Planck equation. We also present several
simulation results, which show that compared with the traditional $L^2$ loss
function, the proposed loss function guides the neural network to significantly
faster convergence. Moreover, we provide empirical evidence that shows that the
proposed loss function, together with the iterative sampling techniques,
performs better in solving high dimensional PDEs.
