  When engineers train deep learning models, they are very much "flying blind".
Commonly used approaches for real-time training diagnostics, such as monitoring
the train/test loss, are limited. Assessing a network's training process solely
through these performance indicators is akin to debugging software without
access to internal states through a debugger. To address this, we present
Cockpit, a collection of instruments that enable a closer look into the inner
workings of a learning machine, and a more informative and meaningful status
report for practitioners. It facilitates the identification of learning phases
and failure modes, like ill-chosen hyperparameters. These instruments leverage
novel higher-order information about the gradient distribution and curvature,
which has only recently become efficiently accessible. We believe that such a
debugging tool, which we open-source for PyTorch, represents an important step
to improve troubleshooting the training process, reveal new insights, and help
develop novel methods and heuristics.
