  Training artificial neural networks requires the optimization of highly
non-convex loss functions. Throughout the years, the scientific community has
developed an extensive set of tools and architectures that render this
optimization task tractable and a general intuition has been developed for
choosing hyper parameters that help the models reach minima that generalize
well to unseen data. However, for the most part, the difference in trainability
in between architectures, tasks and even the gap in network generalization
abilities still remain unexplained. Visualization tools have played a key role
in uncovering key geometric characteristics of the loss-landscape of ANNs and
how they impact trainability and generalization capabilities. However, most
visualizations methods proposed so far have been relatively limited in their
capabilities since they are of linear nature and only capture features in a
limited number of dimensions. We propose the use of the modern dimensionality
reduction method PHATE which represents the SOTA in terms of capturing both
global and local structures of high-dimensional data. We apply this method to
visualize the loss landscape during and after training. Our visualizations
reveal differences in training trajectories and generalization capabilities
when used to make comparisons between optimization methods, initializations,
architectures, and datasets. Given this success we anticipate this method to be
used in making informed choices about these aspects of neural networks.
