  We propose a simple, data-driven approach to help guide hyperparameter
selection for neural network initialization. We leverage the relationship
between neural network and Gaussian process models having corresponding
activation and covariance functions to infer the hyperparameter values
desirable for model initialization. Our experiment shows that marginal
likelihood maximization provides recommendations that yield near-optimal
prediction performance on MNIST classification task under experiment
constraints. Furthermore, our empirical results indicate consistency in the
proposed technique, suggesting that computation cost for the procedure could be
significantly reduced with smaller training sets.
