  In recent years, artificial neural networks have developed into a powerful
tool for dealing with a multitude of problems for which classical solution
approaches reach their limits. However, it is still unclear why randomly
initialized gradient descent optimization algorithms, such as the well-known
batch gradient descent, are able to achieve zero training loss in many
situations even though the objective function is non-convex and non-smooth. One
of the most promising approaches to solving this problem in the field of
supervised learning is the analysis of gradient descent optimization in the
so-called overparameterized regime. In this article we provide a further
contribution to this area of research by considering overparameterized
fully-connected rectified artificial neural networks with biases. Specifically,
we show that for a fixed number of training data the mean squared error using
batch gradient descent optimization applied to such a randomly initialized
artificial neural network converges to zero at a linear convergence rate as
long as the width of the artificial neural network is large enough, the
learning rate is small enough, and the training input data are pairwise
linearly independent.
