  The ability to train randomly initialised deep neural networks is known to
depend strongly on the variance of the weight matrices and biases as well as
the choice of nonlinear activation. Here we complement the existing geometric
analysis of this phenomenon with an information theoretic alternative. Lower
bounds are derived for the mutual information between an input and hidden layer
outputs. Using a mean field analysis we are able to provide analytic lower
bounds as functions of network weight and bias variances as well as the choice
of nonlinear activation. These results show that initialisations known to be
optimal from a training point of view are also superior from a mutual
information perspective.
