  Overfitting is one of the fundamental challenges when training convolutional
neural networks and is usually identified by a diverging training and test
loss. The underlying dynamics of how the flow of activations induce overfitting
is however poorly understood. In this study we introduce a perplexity-based
sparsity definition to derive and visualise layer-wise activation measures.
These novel explainable AI strategies reveal a surprising relationship between
activation sparsity and overfitting, namely an increase in sparsity in the
feature extraction layers shortly before the test loss starts rising. This
tendency is preserved across network architectures and reguralisation
strategies so that our measures can be used as a reliable indicator for
overfitting while decoupling the network's generalisation capabilities from its
loss-based definition. Moreover, our differentiable sparsity formulation can be
used to explicitly penalise the emergence of sparsity during training so that
the impact of reduced sparsity on overfitting can be studied in real-time.
Applying this penalty and analysing activation sparsity for well known
regularisers and in common network architectures supports the hypothesis that
reduced activation sparsity can effectively improve the generalisation and
classification performance. In line with other recent work on this topic, our
methods reveal novel insights into the contradicting concepts of activation
sparsity and network capacity by demonstrating that dense activations can
enable discriminative feature learning while efficiently exploiting the
capacity of deep models without suffering from overfitting, even when trained
excessively.
