N EURAL P ERSISTENCE : A C OMPLEXITY M EASURE FOR D EEP N EURAL N ETWORKS U SING A LGEBRAIC T OPOLOGY
These authors contributed equally
A BSTRACT
While many approaches to make neural networks more fathomable have been proposed, they are restricted to interrogating the network with input data.
Measures for characterizing and monitoring structural properties, however, have not been developed.
In this work, we propose neural persistence , a complexity measure for neural network architectures based on topological data analysis on weighted stratiﬁed graphs.
To demonstrate the usefulness of our approach, we show that neural persistence reﬂects best practices developed in the deep learning community such as dropout and batch normalization.
Moreover, we derive a neural persistencebased stopping criterion that shortens the training process while achieving comparable accuracies as early stopping based on validation loss.
I NTRODUCTION
The practical successes of deep learning in various ﬁelds such as image processing (Simonyan &
Zisserman, 2015; He et al., 2016; Hu et al., 2018), biomedicine (Ching et al., 2018; Rajpurkar et al., 2017; Rajkomar et al., 2018), and language translation (Bahdanau et al., 2015; Sutskever et al., 2014; Wu et al., 2016) still outpace our theoretical understanding.
While hyperparameter adjustment strategies exist (Bengio, 2012), formal measures for assessing the generalization capabilities of deep neural networks have yet to be identiﬁed (Zhang et al., 2017).
Previous approaches for improving theoretical and practical comprehension focus on interrogating networks with input data.
These methods include i) feature visualization of deep convolutional neural networks (Zeiler & Fergus, 2014; Springenberg et al., 2015), ii) sensitivity and relevance analysis of features (Montavon et al., 2017), iii) a descriptive analysis of the training process based on information theory (Tishby &
Zaslavsky, 2015; Shwartz-Ziv & Tishby, 2017; Saxe et al., 2018; Achille & Soatto, 2018), and iv) a statistical analysis of interactions of the learned weights (Tsang et al., 2018).
Additionally,
Raghu et al.
(2017) develop a measure of expressivity of a neural network and use it to explore the empirical success of batch normalization, as well as for the deﬁnition of a new regularization method.
They note that one key challenge remains, namely to provide meaningful insights while maintaining theoretical generality.
This paper presents a method for elucidating neural networks in light of both aspects.
We develop neural persistence , a novel measure for characterizing neural network structural complexity.
In doing so, we adopt a new perspective that integrates both network weights and connectivity while not relying on interrogating networks through input data.
Neural persistence builds on computational techniques from algebraic topology, speciﬁcally topological data analysis (TDA), which was already shown to be beneﬁcial for feature extraction in deep learning (Hofer et al., 2017) and describing the complexity of GAN sample spaces (Khrulkov & Oseledets, 2018).
More precisely, we rephrase deep networks with fully-connected layers into the language of algebraic topology and develop a measure for assessing the structural complexity of i) individual layers, and ii) the entire network.
In this work, we present the following contributions:
- We introduce neural persistence , a novel measure for characterizing the structural complexity of neural networks that can be efﬁciently computed.
- We prove its theoretical properties, such as upper and lower bounds, thereby arriving at a normalization for comparing neural networks of varying sizes. - We demonstrate the practical utility of neural persistence in
two
scenarios:
i) it correctly captures the beneﬁts of dropout and batch normalization during the training process, and
ii
) it can be
easily
used
as
a competitive early stopping criterion that does not
require validation
data.  B ACKGROUND : T OPOLOGICAL DATA ANALYSIS
Topological data analysis (TDA) recently emerged as a ﬁeld that provides computational tools for analysing complex data within a rigorous mathematical framework that is based on algebraic topology .
This paper uses persistent homology, a theory that was developed to understand highdimensional manifolds (Edelsbrunner et al., 2002; Edelsbrunner & Harer, 2010), and has since been successfully employed in characterizing graphs (Sizemore et al., 2017; Rieck et al., 2018), ﬁnding relevant features in unstructured data (Lum et al., 2013), and analysing image manifolds (Carlsson et al., 2008).
This section gives a brief summary of the key concepts; please refer to Edelsbrunner & Harer (2010) for an extensive introduction.
Simplicial homology
The central object in algebraic topology is a simplicial complex K , i.e. a high-dimensional generalization of a graph, which is typically used to describe complex objects such as manifolds.
Various notions to describe the connectivity of K exist, one of them being simplicial homology.
Brieﬂy put, simplicial homology uses matrix reduction algorithms (Munkres, 1996) to derive a set of groups, the homology groups, for a given simplicial complex K .
Homology groups describe topological features—colloquially also referred to as holes—of a certain dimension d , such as connected components ( d = 0 ), tunnels ( d = 1 ), and voids ( d = 2 ).
The information from the d th homology group is summarized in a simple complexity measure, the d th Betti number (cid:12) d , which merely counts the number of d -dimensional features: a circle, for example, has Betti numbers (1 ; 1) , i.e. one connected component and one tunnel, while a ﬁlled circle has Betti numbers (1 ; 0) , i.e. one connected component but no tunnel.
In the context of analysing simple feedforward neural networks for two classes, Bianchini & Scarselli (2014) calculated bounds of Betti numbers of the decision region belonging to the positive class, and were thus able to show the implications of different activation functions.
These ideas were extended by Guss & Salakhutdinov (2018) to obtain a measure of the topological complexity of decision boundaries.
Persistent homology For the analysis of real-world data sets, however, Betti numbers turn out to be of limited use because their representation is too coarse and unstable.
This prompted the development of persistent homology.
Given a simplicial complex K with an additional set of weights a 0 (cid:20) a 1 (cid:20) (cid:1) (cid:1) (cid:1) (cid:20) a m (cid:0) 1 (cid:20) a m , which are commonly thought to represent the idea of a scale, it is possible to put K in a ﬁltration, i.e. a nested sequence of simplicial complexes ∅ =
K 0 (cid:18) K 1 (cid:18) (cid:1) (cid:1) (cid:1) (cid:18) K m (cid:0) 1 (cid:18) K m = K .
This ﬁltration is thought to represent the ‘growth’ of K as the scale is being changed.
During this growth process, topological features can be created (new vertices may be added, for example, which creates a new connected component) or destroyed (two connected components may merge into one).
Persistent homology tracks these changes and represents the creation and destruction of a feature as a point ( a i ; a j ) 2 R 2 for indices i (cid:20) j with respect to the ﬁltration.
The collection of all points corresponding to d -dimensional topological features is called the d th persistence diagram D d .
It can be seen as a collection of Betti numbers at multiple scales.
Given a point ( x; y ) 2 D d , the quantity pers( x; y ) :
= j y (cid:0) x j is referred to as its persistence .
Typically, high persistence is considered to correspond to features, while low persistence is considered to indicate noise (Edelsbrunner et al., 2002).
A NOVEL MEASURE FOR NEURAL NETWORK COMPLEXITY
This section details neural persistence , our novel measure for assessing the structural complexity of neural networks.
By exploiting both network structure and weight information through persistent homology, our measure captures network expressiveness and goes beyond mere connectivity properties.
Subsequently, we describe its calculation, provide theorems for theoretical and empirical
w
[Figure]
w
[Figure]
w
[Figure]
(cid:18) (cid:18) llllllw ′ d w ′ c bounds, and show the existence of neural networks complexity regimes.
To summarize this section, Figure 1 illustrates how our method treats a neural network.
3. N EURAL PERSISTENCE
Given a feedforward neural network with an arrangement of neurons and their connections E , let W refer to the set of weights.
Since W is typically changing during training, we require a function φ : E !
W that maps a speciﬁc edge to a weight.
Fixing an activation function, the connections form a stratiﬁed graph .
Deﬁnition 1 (Stratiﬁed graph and layers) .
A stratiﬁed graph is a multipartite graph G =
( V; E ) satisfying V
= V
⊔ V 1 ⊔ : : : , such that if u 2 V i , v 2 V j , and ( u; v ) 2 E , we have j =
i
+ 1 .
Hence, edges are only permitted between adjacent vertex sets.
Given k 2 N , the k th layer of a stratiﬁed graph is the unique subgraph G k :=
( V k ⊔ V k +1 ; E k :=
E \ f V k (cid:2)
V k +1 g ) .
This enables calculating the persistent homology of G and each G k , using the ﬁltration induced by sorting all weights, which is common practice in topology-based network analysis (Carstens & Horadam, 2013; Horak et al., 2009) where weights often represent closeness or node similarity.
However, our context requires a novel ﬁltration because the weights arise from an incremental ﬁtting procedure, namely the training, which could theoretically lead to unbounded values.
When analysing geometrical data with persistent homology, one typically selects a ﬁltration based on the (Euclidean) distance between data points (Bubenik, 2015).
The ﬁltration then connects points that are increasingly distant from each other, starting from points that are direct neighbours.
Our network ﬁltration aims to mimic this behaviour in the context of fully-connected neural networks.
Our framework does not explicitly take activation functions into account; however, activation functions inﬂuence the evolution of weights during training.
Filtration Given the set of weights W for one training step, let w max := max w 2W j w j .
Furthermore, let W ′ :
=
fj w j =w max
j w
Wg be the set of transformed weights, indexed in non-ascending order, such that 1 = w ′ 0 (cid:21) w ′ 1 (cid:21) (cid:1) (cid:1) (cid:1) (cid:21) 0 .
This permits us to deﬁne a ﬁltration for the k th layer
G k as G (0) k (cid:18) G (1) k
(cid:18) : : : , where G ( i ) k :=
( V k ⊔ V k +1 ; f ( u; v ) j
( u; v ) 2 E k ^ φ ′ ( u; v ) (cid:21) w ′
i g ) and φ ′ ( u; v ) 2 W ′ denotes the transformed weight of an edge.
We tailored this ﬁltration towards the analysis of neural networks, for which large (absolute) weights indicate that certain neurons exert a larger inﬂuence over the ﬁnal activation of a layer.
The strength of a connection is thus preserved by the ﬁltration, and weaker weights with j w j (cid:25) 0 remain close to 0 .
Moreover, since w ′ 2 [0 ; 1] holds for the transformed weights, this ﬁltration makes the network invariant to scaling, which simpliﬁes the comparison of different networks.
Persistence diagrams Having set up the ﬁltration, we can calculate persistent homology for every layer G k .
As the ﬁltration contains at most 1 -simplices (edges), we capture zero-dimensional topological information, i.e. how connected components are created and merged during the ﬁltration.
These information are structurally equivalent to calculating a maximum spanning tree using the
Require: Neural network with l layers and weights W 1: w max max w 2W j w j ▷ Determine largest absolute weight 2:
W ′ fj
w j =w max j w
Wg ▷ Transform weights for ﬁltration 3: for k 2 f 0 ; : : : ; l (cid:0) 1 g do 4:
F k
G (0) k (cid:18) G (1) k (cid:18) : : : ▷ Establish ﬁltration of k th layer 5:
D k
P ERSISTENT H OMOLOGY ( F k )
▷ Calculate persistence diagram 6: end for 7: return f∥D 0 ∥ p ; : : : ; ∥D l (cid:0) 1 ∥
p g ▷ Calculate neural persistence for each layer weights, or performing hierarchical clustering with a speciﬁc setup (Carlsson & Mémoli, 2010). While it would theoretically be possible to include higher-dimensional information about each layer G k , for example in the form of cliques (Rieck et al., 2018), we focus on zero-dimensional information in this paper, because of the following advantages: i) the resulting values are easily interpretable as they essentially describe the clustering of the network at multiple weight thresholds, ii) previous research (Rieck & Leitte, 2016; Hofer et al., 2017) indicates that zero-dimensional topological information is already capturing a large amount of information, and iii) persistent homology calculations are highly efﬁcient in this regime (see below).
We thus calculate zero-dimensional persistent homology with this ﬁltration.
The resulting persistence diagrams have a special structure: since our ﬁltration solely sorts edges , all vertices are present at the beginning of the ﬁltration, i.e. they are already part of G (0) k for each k .
As a consequence, they are assigned a weight of 1 , resulting in j
V k (cid:2) V k +1 j connected components.
Hence, entries in the corresponding persistence diagram D k are of the form (1 ; x ) , with x 2 W ′ , and will be situated below the diagonal, similar to superlevel set ﬁltrations (Bubenik, 2015; Cohen-Steiner et al., 2009).
Using the p -norm of a persistence diagram, as introduced by Cohen-Steiner et al.
(2010), we obtain the following deﬁnition for neural persistence.
Deﬁnition 2 (Neural persistence) .
The neural persistence of the k th layer G k , denoted by NP( G k ) , is the p -norm of the persistence diagram D k resulting from our previously-introduced ﬁltration, i.e.
[Equation]
which (for p = 2 ) captures the Euclidean distance of points in D k to the diagonal.
The p -norm is known to be a stable summary (Cohen-Steiner et al., 2010) of topological features in a persistence diagram.
For neural persistence to be a meaningful measure of structural complexity, it should increase as a neural network is learning.
We evaluate this and other properties in Section 4.
Algorithm 1 provides pseudocode for the calculation process.
It is highly efﬁcient: the ﬁltration (line 4) amounts to sorting all n weights of a network, which has a computational complexity of O ( n log n ) .
Calculating persistent homology of this ﬁltration (line 5) can be realized using an algorithm based on union–ﬁnd data structures Edelsbrunner et al.
This has a computational complexity of O ( n (cid:1) (cid:11) ( n )) , where (cid:11) ( (cid:1) ) refers to the extremely slow-growing inverse of the Ackermann function (Cormen et al., 2009, Chapter 22).
We make our implementation and experiments available under https://github.com/BorgwardtLab/Neural-Persistence .
3. P ROPERTIES OF NEURAL PERSISTENCE
We elucidate properties about neural persistence to permit the comparison of networks with different architectures.
As a ﬁrst step, we derive bounds for the neural persistence of a single layer G k .
Theorem 1.
Let G k be a layer of a neural network according to Deﬁnition 1.
Furthermore, let φ k : E k !
W ′ denote the function that assigns each edge of G k a transformed weight.
Using the ﬁltration from Section 3.1 to calculate persistent homology, the neural persistence NP( G k ) of the k th layer satisﬁes
[Equation]
where j V k (cid:2)
V k +
j denotes the cardinality of the vertex set, i.e. the number of neurons in the layer.
Proof.
We prove this constructively and show that the bounds can be realized.
For the lower bound, let G (cid:0) k be a fully-connected layer with j V k j vertices and, given (cid:18) 2 [0 ; 1] , let φ k ( e ) :
=
(cid:18) for every edge e .
Since a vertex v is created before its incident edges, the ﬁltration degenerates to a lexicographical ordering of vertices and edges, and all points in D k will be of the form ( (cid:18); (cid:18) ) .
Thus,
NP( G (cid:0) k ) = 0 .
For the upper bound, let G + k again be a fully-connected layer with j V k j (cid:21) 3 vertices and let a; b 2 [0 ; 1] with a < b .
Select one edge e ′ at random and deﬁne a weight function as φ ( e ′ ) :
=
b and φ ( e ) :
= a otherwise.
In the ﬁltration, the addition of the ﬁrst edge will create a pair of the form ( b; b ) , while all other pairs will be of the form ( b; a ) .
Consequently, we have
[Equation]
so our upper bound can be realized.
To show that this term cannot be exceeded by NP( G ) for any
G , suppose we perturb the weight function e φ ( e ) :
= φ ( e ) + ϵ 2 [0 ; 1] .
This cannot increase NP , however, because each difference b (cid:0) a in Equation 3 is maximized by max φ ( e ) (cid:0) min φ ( e ) .
We can use the upper bound of Theorem 1 to normalize the neural persistence of a layer, making it possible to compare layers (and neural networks) that feature different architectures, i.e. a different number of neurons.
Deﬁnition 3 (Normalized neural persistence) .
For a layer G k following Deﬁnition 1, using the upper bound of Theorem 1, the normalized neural persistence f NP( G k ) is deﬁned as the neural persistence of G k divided by its upper bound, i.e. f NP
[Equation]
G k
[Equation]
[Equation]
The normalized neural persistence of a layer permits us to extend the deﬁnition to an entire network.
While this is more complex than using a single ﬁltration for a neural network, this permits us to side-step the problem of different layers having different scales.
Deﬁnition 4 (Mean normalized neural persistence) .
Considering a network as a stratiﬁed graph
G according to Deﬁnition 1, we sum the neural persistence values per layer to obtain the mean normalized neural persistence , i.e.
[Equation]
While Theorem 1 gives a lower and upper bound in a general setting, it is possible to obtain empirical bounds when we consider the tuples that result from the computation of a persistence diagram.
Recall that our ﬁltration ensures that the persistence diagram of a layer contains tuples of the form (1 ; w i ) , with w i 2 [0 ; 1] being a transformed weight.
Exploiting this structure permits us to obtain bounds that could be used prior to calculating the actual neural persistence value in order to make the implementation more efﬁcient.
Theorem 2.
Let G k be a layer of a neural network as in Theorem 1 with n vertices and m edges whose edge weights are sorted in non-descending order, i.e. w 0 (cid:20) w 2 (cid:20) (cid:1) (cid:1) (cid:1) (cid:20) w m (cid:0) 1 .
Then
NP( G k ) can be empirically bounded by
[Equation]
) where w max = ( w m (cid:0) 1 ; w m (cid:0) 2 ; : : : ; w m (cid:0) n ) T and w min = ( w 0 ; w 2 ; : : : ; w n (cid:0) 1 ) T are the vectors containing the n largest and n smallest weights, respectively.
Proof.
See Section A.2 in the appendix.
Complexity regimes in neural persistence As an application of the two theorems, we brieﬂy take a look at how neural persistence changes for different classes of simple neural networks.
To this end, we train a perceptron on the ‘MNIST’ data set.
Since our measure uses the weight matrix of a perceptron, we can compare its neural persistence with the neural persistence of random weight matrices, drawn from different distributions.
Moreover, we can compare trained networks with respect to their initial parameters.
Figure 2 depicts the neural persistence values as well as the lower bounds according to Theorem 2 for different settings.
We can see that a network in which the optimizer diverges (due to improperly selected parameters) is similar to a random Gaussian matrix. [Figure]
Trained networks, on the other hand, are clearly distinguished from all other networks.
Uniform matrices have a signiﬁcantly lower neural persistence than Gaussian ones.
This is in line with the intuition that the latter type of networks induces functional sparsity because few neurons have large absolute weights.
For clarity, we refrain from showing the empirical upper bounds because most weight distributions are highly right-tailed; the bound will not be as tight as the lower bound.
These results are in line with a previous analysis (Sizemore et al., 2017) of small weighted networks, in which persistent homology is seen to outperform traditional graph-theoretical complexity measures such as the clustering coefﬁcient (see also Section A.1 in the appendix).
For deeper networks, additional experiments discuss the relation between validation accuracy and neural persistence (Section A.5), the impact of different data distributions, as well as the variability of neural persistence for architectures of varying depth (Section A.6).
E XPERIMENTS
This section demonstrates the utility and relevance of neural persistence for fully connected deep neural networks.
We examine how commonly used regularization techniques (batch normalization and dropout) affect neural persistence of trained networks.
Furthermore, we develop an early stopping criterion based on neural persistence and we compare it to the traditional criterion based on validation loss.
We used different architectures with ReLU activation functions across experiments.
The brackets denote the number of units per hidden layer.
In addition, the Adam optimizer with hyperparameters tuned via cross-validation was used unless noted otherwise.
Please refer to Table A. in the appendix for further details about the experiments.
4. D EEP LEARNING BEST
We compare the mean normalized neural persistence (see Deﬁnition 4) of a two-layer (with an architecture of [650 ; 650] ) neural network to two models where batch normalization (Ioffe & Szegedy, 2015) or dropout (Srivastava et al., 2014) are applied.
Figure 3 shows that the networks designed according to best practices yield higher normalized neural persistence values on the ‘MNIST’ data set [-0 :
-]in comparison to an unmodiﬁed network.
{+The effect of dropout on the mean normalized neural persistence is more pronounced and this trend is directly analogous to the observed accuracy on 0 :0 :
+}
[Figure]
the test set.
These results are consistent with expectations if we consider dropout to be similar to ensemble learning (Hara et al., 2016).
As individual parts of the network are trained independently, a higher degree of per-layer redundancy is expected, resulting in a different structural complexity.
Overall, these results indicate that for a ﬁxed architecture approaches targeted at increasing the neural persistence during the training process may be of particular interest.
4. E ARLY STOPPING BASED ON NEURAL PERSISTENCE
Neural persistence can be used as an early stopping criterion that does not require a validation data set to prevent overﬁtting: if the mean normalized neural persistence does not increase by more than ∆ min during a certain number of epochs g , the training process is stopped.
This procedure is called ‘patience’ and Algorithm 2 describes it in detail.
A similar variant of this algorithm, using validation loss instead of persistence, is the state-of-the-art for early stopping in training (Bengio, 2012; Chollet et al., 2015).
To evaluate the efﬁcacy of our measure, we compare it against validation loss in an extensive set of scenarios.
More precisely, for a training process with at most G epochs, we deﬁne a G (cid:2) G parameter grid consisting of the ‘patience’ parameter g and a burn-in rate b (both measured in epochs).
b deﬁnes the number of epochs after which an early stopping criterion starts monitoring, thereby preventing underﬁtting.
Subsequently, we set ∆ min = 0 for all measures to remain comparable and scale-invariant, as non-zero values could implicitly favour one of them due to scaling.
For each data set, we perform 100 training runs of the same architecture, monitoring validation loss and mean normalized neural persistence every quarter epoch.
The early stopping behaviour of both measures is simulated for each combination of b and g and their performance over all runs is summarized in terms of median test accuracy and median stopping epoch; if a criterion is not triggered for one run, we report the test accuracy at the end of the training and the number of training epochs.
This results in a scatterplot, where each point (corresponding to a single parameter combination) shows the difference in epochs and the absolute difference in test accuracy (measured in percent).
The quadrants permit an intuitive explanation: Q 2 , for example, contains all conﬁgurations for which our measure stops earlier , while achieving a higher accuracy.
Since b and g are typically chosen to be small in an early stopping scenario, we use grey points to indicate uncommon conﬁgurations for which b or g is larger than half of the total number of epochs.
Furthermore, to summarize the performance of our measure, we calculate the barycentre of all conﬁgurations (green square).
Figure 4a depicts the comparison with validation loss for the ‘Fashion-MNIST’ (Xiao et al., 2017) data set; please refer to Section A.3 in the appendix for more data sets.
Here, we observe that most common conﬁgurations are in Q 2 or in Q 3 , i.e our criterion stops earlier.
The barycentre is at ( (cid:0) 0 : 53 ; (cid:0) 0 : 08) , showing that out of 625 conﬁgurations, on average we stop half an epoch earlier than validation loss, while losing virtually no accuracy ( 0 : 08% ).
Figure 4c depicts detailed differences in accuracy and epoch for our measure when compared to validation loss; each cell in a heatmap corresponds to a single parameter conﬁguration of b and g .
In the heatmap of accuracy differences, blue, white, and red represent parameter combinations for which we obtain higher, equal, or lower accuracy, respectively, than with validation loss for the same parameters. Similarly, in the
Algorithm 2 Early stopping based on mean normalized neural persistence
Require:
Weighted neural network N , patience g , ∆ min 1: P 0 , G ▷ Initialize highest observed value and patience counter 2: procedure E ARLY S TOPPING
( N , g , ∆ min ) ▷ Callback that monitors training at every epoch 3:
P ′
NP( N ) 4: if P ′ > P + ∆ min then ▷ Update mean normalized neural persistence and reset counter 5:
P
P ′ , G 6: else ▷ Update patience counter 7:
G
G +8: end if 9: if G (cid:21) g then ▷ Patience criterion has been triggered 10: return P ▷ Stop training and return highest observed value 11: end if 12: end procedure
[Figure]
heatmap of epoch differences, green represents parameter combinations for which we stop earlier than validation loss.
For b (cid:20) 8 , we stop earlier ( 0 : 62 epochs on average), while losing only 0 : 06% accuracy.
Finally, Figure 4d shows how often each measure is triggered.
Ideally, each measure should consist of a dark green triangle, as this would indicate that each conﬁguration stops all the time.
For this data set, we observe that our method stops for more parameter combinations than validation loss, but not as frequently for all of them.
To ensure comparability across scenarios, we did not use the validation data as additional training data when stopping with neural persistence; we refer to Section A.7 for additional experiments in data scarcity scenarios.
We observe that our method stops earlier when overﬁtting can occur, and it stops later when longer training is beneﬁcial.
D ISCUSSION
In this work, we presented neural persistence , a novel topological measure of the structural complexity of deep neural networks.
We showed that this measure captures topological information that pertains to deep learning performance.
Being rooted in a rich body of research, our measure is theoretically well-deﬁned and, in contrast to previous work, generally applicable as well as computationally efﬁcient.
We showed that our measure correctly identiﬁes networks that employ best practices such as dropout and batch normalization.
Moreover, we developed an early stopping criterion that exhibits competitive performance while not relying on a separate validation data set.
Thus, by saving valuable data for training, we managed to boost accuracy, which can be crucial for enabling deep learning in regimes of smaller sample sizes.
Following Theorem 2, we also experimented with using the p -norm of all weights of the neural network as a proxy for neural persistence.
However, this did not yield an early stopping measure because it was never triggered, thereby suggesting that neural persistence captures salient information that would otherwise be hidden among all the weights of a network.
We extended our framework to convolutional neural networks (see Section A.4) by deriving a closed-form approximation, and observed that an early stopping criterion based on neural persistence for convolutional layers will require additional work.
Furthermore, we conjecture that assessing dissimilarities of networks by means of persistence diagrams (making use of higher-dimensional topological features), for example, will lead to further insights regarding their generalization and learning abilities.
Another interesting avenue for future research would concern the analysis of the ‘function space’ learned by a neural network.
On a more general level, neural persistence demonstrates the great potential of topological data analysis in machine learning.
persistence
A A PPENDIX A. C OMPARISON WITH GRAPH - THEORETICAL MEASURES
Traditional complexity/structural measures from graph theory, such as the clustering coefﬁcient, the average shortest path length, and global/local efﬁciency are already known to be insufﬁciently accurate to characterize different models of complex random networks Sizemore et al.
Our experiments indicate that this holds true for (deep) neural networks, too.
As a brief example, we trained a perceptron on the MNIST data set with batch stochastic gradient descent ( (cid:17) = 0 : 5 ), achieving a test accuracy of (cid:25) 0 : 91 .
Moreover, we intentionally ‘sabotaged’ the training by setting (cid:17) = 1 (cid:2) 10 (cid:0)such that SGD is unable to converge properly.
This leads to networks with accuracies ranging from 0 : 38 – 0 : 65 .
A complexity measure should be capable of distinguishing both classes of networks.
However, as Figure A.1 (top) shows, this is not the case for the clustering coefﬁcient.
Neural persistence (bottom), on the other hand, results in two regimes that can clearly be distinguished, with the trained networks having a signiﬁcantly smaller variance.
A. P ROOF OF T HEOREM
Proof.
We may consider the ﬁltration from Section 3.1 to be a subset selection problem with constraints, where we select n out of m weights.
The neural persistence NP( G k ) of a layer thus only depends on the selected weights that appear as tuples of the form (1 ; w i ) in D k .
Letting e w denote the vector of selected weights arising from the persistence diagram calculation, we can rewrite neural persistence as NP( G k ) =
∥ 1 (cid:0) e w ∥ p .
Furthermore, e w satisﬁes ∥ w min ∥ p (cid:20) ∥ e w ∥ p (cid:20) ∥
w max ∥ p .
Since all transformed weights are non-negative in our ﬁltration, it follows that (note the reversal of the two terms)
[Equation]
and the claim follows.
A. A DDITIONAL VISUALIZATIONS AND ANALYSES FOR EARLY STOPPING
Due to space constraints and the large number of conﬁgurations that we investigated for our early stopping experiments, this section contains additional plots that follow the same schematic: the top row shows the differences in accuracy and epoch for our measure when compared to the commonlyused validation loss.
Each cell in the heatmap corresponds to a single conﬁguration of b and g .
In the heatmap of accuracy differences, blue represents parameter combinations for which we obtain a higher accuracy than validation loss for the same parameters; white indicates combinations for which we obtain the same accuracy, while red highlights combinations in which our accuracy decreases.
Similarly, in the heatmap of epoch differences, green represents parameter combinations for which we stop earlier than validation loss for the same parameter.
The scatterplots in Section 4.2 show an ‘unrolled’ version of this heat map, making it possible to count how many parameter combinations result in early stops while also increasing accuracy, for example.
The heatmaps, by contrast, make it possible to compare the behaviour of the two measures with respect to each parameter combination.
Finally, the bottom row of every plot shows how many times each measure was triggered for every parameter combination.
We consider a measure to be triggered if its stopping condition is satisﬁed prior to the last training epoch.
Due to the way the parameter grid is set up, no conﬁguration above the diagonal can stop, because b + g would be larger than the total number of training epochs.
This permits us to compare the ‘slopes’ of cells for each measure.
Ideally, each measure should consist of a dark green triangle, as this would indicate that parameter conﬁguration stops all the time.
MNIST Please refer to Figures A.2 and A.3.
The colours in the difference matrix of the top row are slightly skewed because in a certain conﬁguration, our measure loses 0 : 8% of accuracy when stopping.
However, there are many other conﬁgurations in which virtually no accuracy is lost and in which we are able to stop more than four epochs earlier.
The heatmaps in the bottom row again indicate that neural persistence is capable of stopping for more parameter combinations in general.
We do not trigger as often for some of them, though.
CIFAR-10 Please refer to Figure A.4.
In general, we observe that this data set is more sensitive with respect to the parameters for early stopping.
While there are several conﬁgurations in which neural persistence stops with an increase of almost 10% in accuracy, there are also scenarios in which we cannot stop training earlier, or have to train longer (up to 15 epochs out of 80 epochs in total).
The second row of plots shows our measure triggers reliably for more conﬁgurations than validation loss.
Overall, the scatterplot of all scenarios (Figure A.5) shows that most practical conﬁgurations are again located in Q 2 and Q 3 .
While we may thus ﬁnd certain conﬁgurations in which we reliably outperform validation loss as an early stopping criterion, we also want to point out that our measures behaves correctly for many practical conﬁgurations.
Points in Q 1 , where we train longer and achieve a higher accuracy, are characterized by a high patience g of approximately 40 epochs and a low burn-in rate b , or vice versa.
This is caused by the training for CIFAR-10, which does not reliably converge for FCNs.
Figure A.6 demonstrates this by showing loss curves and the mean normalized neural persistence curves of ﬁve runs over training (loss curves have been averaged over all runs; standard deviations are shown in grey; we show the ﬁrst half of the training to highlight the behaviour for practical early stopping conditions).
For ‘Fashion-MNIST’, we observe that NP exhibits clear change points during the training process, which can be exploited for early stopping.
For ‘CIFAR-10’, we observe a rather incremental growth for some runs (with no clearlydeﬁned maximum), making it harder to derive a generic early stopping criterion that does not depend on ﬁne-tuned parameters.
Hence, we hypothesize that neural persistence cannot be used reliably in scenarios where the architecture is incapable of learning the data set.
In the future, we plan to experiment with deliberately selected ‘bad’ and ‘good’ architectures in order to evaluate to what extent our topological measure is capable of assessing their suitability for training, but this is beyond the scope of this paper.
IMDB Please refer to Figure A.7.
For this data set, we observe that most parameter conﬁgurations result in earlier stopping (up to two epochs earlier than validation loss), with accuracy increases of up to 0 : 10% .
This is also shown in the scatterplot A.8.
Only a single conﬁguration, viz.
g =and b = 0 , results in a severe loss of accuracy; we removed it from the scatterplot for reasons of clarity, as its accuracy difference of (cid:0) 21% would skew the display of the remaining conﬁgurations too much (this is also why the legends do not include this outlier).
[Figure]
[Figure]
difference −0.8−0.
−0.4−0.
12 16 20 24 g      b
{+Epoch difference −6 − −
+}
[Figure] [Figure]
NP
[Figure]
b
Validation loss
[Figure] [Figure]
{+(cid:0) 0 :(cid:0) 0 : 
+}
[Figure]
[Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure]
A. N EURAL P ERSISTENCE FOR C ONVOLUTIONAL L AYERS
In principle, the proposed ﬁltration process could be applied to any bipartite graph.
Hence, we can directly apply our framework to convolutional layers, provided we represent them properly.
Speciﬁcally, for layer l we represent the convolution of its i th input feature map a ( l (cid:0) 1)
i {+
+}
R h in (cid:2) w in with the j th ﬁlter H j 2 R p (cid:2) q as one bipartite graph G i;j parametrized by a sparse weight matrix
W ( l ) i;j 2 R ( h out (cid:1) w out ) (cid:2) ( h in (cid:1) w in ) , which in each row contains the p (cid:1) q unrolled values of H j on the diagonal, with h in (cid:0) p zeros padded in between after each p values of vec ( H j ) .
This way, the ﬂattened pre-activation can be described as vec ( z ( l ) i;j ) =
W ( l ) i;j (cid:1) vec ( a ( l (cid:0) 1) i )
+ b li
[Equation]
j (cid:1) 1 ( h out (cid:1) w out ) (cid:2) 1 .
Since ﬂattening does not change the topology of our bipartite graph, we compute the normalized neural persistence on this sparse weight matrix W ( l ) i;j as the unrolled analogue of the fully-connected network’s weight matrix.
Averaging over all ﬁlters then gives a per-layer measure, similar to the way we derived mean normalized neural persistence in the main paper.
When studying the unrolled adjacency matrix W ( l ) i;j , it becomes clear that the edge ﬁltration process can be approximated in a closed form.
Speciﬁcally, for m and n input and output neurons we initialize (cid:28) = m + n connected components.
When using zero padding, the additional dummy input neurons have to included in m .
For all (cid:28) tuples in the persistence diagram the creation event c = 1 .
Notably, each output neuron shares the same set of edge weights.
Due to this, the destruction events—except for a few special cases—simplify to a list of length (cid:28) containing the largest ﬁlter values (each value is contained n times) in descending order until the list is ﬁlled.
This simpliﬁcation of neural persistence of a convolution with one ﬁlter is shown as a closed expression in Equations 7–11, and our implementation is sketched in Algorithm 3. We thus obtain
[Equation]
where we use
[Equation]
with
[Equation]
[Equation]
[-B y = w ⌊ yn ⌋ +
 (cid:10) 1 y mod n with-]where 1 0 := 0 .
Following this notation, Equation 7 expresses neural persistence of the bipartite graph G i;j , with e w denoting the vector of selected weights (i.e. the destruction events) when calculating the persistence diagram.
We use w to denote the ﬂattened and sorted weight values (in descending order) of the convolutional ﬁlter H j , while w c represents the vector of all weights that are located in a corner of H j , whereas w (cid:22) c;ϕ is the vector of all weights which do not originate from the corner of the ﬁlter while still belonging to the ﬁrst (and thus largest ) ⌊ ϕn ⌋ weights in w , which we denote by w 1: ⌊ ϕn ⌋ .
For the subsequent experiments (see below), we use a simple CNN that employs 32 + 2048 ﬁlters.
Hence, by using the shortcut described above, we do not have to unroll 2080 weight matrices explicitly, thereby gaining both in memory efﬁciency and run time, as compared to the naive approach: on average, a naive exact computation based on unrolling required 8 : 77 s per convolutional ﬁlter and evaluation step, whereas the approximation only took about 0 :
38 s while showing very similar behaviour up to a constant offset.
For our experiments, we used an off-the-shelf ‘LeNet-like’ CNN model architecture (two convolutional layers each with max pooling and ReLU, 1 fully-connected and softmax) as described in Abadi et al.
We trained the model on ‘Fashion-MNIST’ and included this setup in the early stopping experiments (100 runs of 20 epochs).
In Figure A.9, we observe that stopping based on the neural persistence of a convolutional layer typically only incurs a considerable loss of accuracy: given a ﬁnal test accuracy of 91 : 73 (cid:6) 0 : 13 , stopping with this naive extension of our measure reduces accuracy by up to 4%.
Furthermore, in contrast to early stopping on a fully-connected architecture,
Require: ﬁlter H 2 R p (cid:2) q ; number of input and output neurons as m; n 1: T ∅ ▷ Initialize set of tuples for persistence diagram 2: (cid:28) m + n; t 0 ; i ▷ Initialize number of tuples, tuple counter, weight index 3: h max max h 2 H
j h j ▷ Determine largest absolute weight 4: H ′ fj h j =h max j h 2 H g ▷ Transform weights for ﬁltration 5: s sort ( vec ( H ′ ))
▷ Sort weights in descending order 6: H ′ c f h ′ 0 ; 0 ; h ′ 0 ;q (cid:0) 1 ; h ′ p (cid:0) 1 ; 0 ; h ′ p (cid:0) 1 ;q (cid:0) 1 g ▷ Determine the set of all corner weights of ﬁlter H ′ 7: T (1 ; 0) ; t t +▷ Add tuple for surviving component 8: for h ′ c 2 H ′ c
do ▷ Each corner of H ′ merges components 9:
T (1 ; h ′ c ) ; t t +10: end for 11: while 1 do ▷ Create the remaining tuples (Approximation step) 12: n ′ = n (cid:0) Ind ( s [ i ] 2 H ′ c )
▷ if current weight is a corner weight, write one less tuple 13: if t + n ′ (cid:20) (cid:28) then ▷ if there are at least n ′ more tuples, set their merge value to s [ i ] 14: repeat n ′ times 15:
T (1 ; s [ i ])
▷ approximative as s [ i ] does not always add n ′ merges due to loops 16: t t + n ′ ;
i i +17: else ▷ otherwise, process the remaining tuples similarly 18: repeat ( (cid:28) (cid:0) t ) times 19:
T (1 ; s [ i ]) 20: break 21: end if 22: end while 23: return ∥T ∥
p ▷ Compute norm of approximated persistence diagram we do not observe any parameter combinations that stop early and increase accuracy.
In fact, there is no conﬁguration that results in an increased accuracy. This empirically conﬁrms our theoretical scepticism towards naively applying our edge-focused ﬁltration scheme to CNNs.
A. R ELATIONSHIP BETWEEN NEURAL PERSISTENCE AND VALIDATION ACCURACY
Motivated by Figure 2, which shows the different ‘regimes’ of neural persistence for a perceptron network, we investigate a possible correlation of (high) neural persistence with (high) predictive accuracy.
For deeper networks, we ﬁnd that neural persistence measures structural properties that arise from different parameters (such as training procedures or initializations), and no correlation can be observed.
For our experiments, we constructed neural networks with a high neural persistence prior to training.
More precisely, following the theorems in this paper, we initialized most weights of each layer with very low values and reserved high values for very few weights.
This was achieved by sampling the weights from a beta distribution with (cid:11) =
: 005 and (cid:12) = 0 : 5 .
Using this procedure, we are able to initialize [20,20,20] networks with NP (cid:25) 0 : 90 (cid:6) 0 : 003 compared to the same networks that have
NP (cid:25) 0 : 38 (cid:6) 0 : 004 when initialized by Xavier initialization.
The mean validation accuracy of these untrained networks on the ‘Fashion-MNIST’ data set is 0 : 10 (cid:6) 0 : 01 and 0 : 09 (cid:6) 0 : 03 , respectively.
Figure A.10 depicts how both types of networks converge to similar regimes of validation accuracy, while the mean normalized neural persistence achieved at the end of the training varies.
For networks initialized with high NP (Figure A.10, left) the validation accuracy of networks with ﬁnal 0 : 9 (cid:20)
NP (cid:20) 0 : 95 ranges from 0 : 098 (not shown) to 0 : 863 .
For Xavier initialization (Figure A.10, right), the lack of correlation can also be observed.
Furthermore, comparing the two plots, there are no clear advantages in initializing networks with high NP .
This observation further motivates the proposed early stopping criterion , which checks for changes in the NP value, and considers stagnating values to be indicative of a trained network.
[Figure]
b
Accuracy difference − −
[Figure]
b
Epoch difference −15−10− (a) Accuracy and epoch differences
[Figure]
Validation loss
[Figure] [Figure] [Figure] r
[Figure]
li z e d N e u r
[Figure]
l P e r s i s t e n c e
A. N EURAL PERSISTENCE FOR DIFFERENT DATA DISTRIBUTIONS AND DEEPER FCN ARCHITECTURES
Neural persistence captures information about different data distributions during training.
The weights tuned via backpropagation are directly inﬂuenced by the input data (as well as their labels) and neural persistence tracks those changes.
To demonstrate this, we trained the same architecture , i.e. [50 ; 50 ; 20] , on two data sets with the same dimensions but different properties: MNIST and ‘Fashion-MNIST’.
Each data set has the same image size ( 28 (cid:2) 28 pixels, one channel) but lay on different manifolds.
Figure A.11 (left) shows a histogram of the mean normalized neural persistence ( NP ) after 25 epochs of training over 100 different runs.
The distributions have a similar shape but are shifted, indicating that the two datasets lead the network to different topological regimes.
We also investigated the effect of depth on neural persistence.
We selected a ﬁxed layer size ( hidden units) and increased the number of hidden layers.
Figure A.11 (right) depicts the boxplots of mean NP for multiple architectures after 15 epochs of training on MNIST.
Adding layers initially increases the variability of NP by enabling the network to converge to different regimes (essentially, there are many more valid conﬁgurations in which a trained neural network might end up in).
However, this effect is reduced after a certain depth: networks with deeper architectures exhibit less variability in NP .
A. E ARLY STOPPING IN DATA SCARCITY SCENARIOS
Labelled data is expensive in most domains of interest, which results in small data sets or low quality of the labels.
We investigate the following experimental set-ups: (1) Reducing the training data set size and (2) Permuting a fraction of the training labels.
We train a fully connected network ( [500 ; 500 ; 200] architecture) on ‘MNIST’ and ‘Fashion-MNIST’.
In the experiments, we compare the following measures for stopping the training: i) Stopping at the optimal test accuracy.
ii)
Fixed stopping after the burn in period.
iii)
Neural persistence patience criterion.
iv) Training loss patience criterion.
v) Validation loss patience criterion.
For a description of the patience criterion, see Algorithm 2.
All measures, except validation loss, include the validation datasets ( 20% ) in the training process to simulate a larger data set when no cross-validation is required.
We report the accuracy on the non-reduced, non-permuted test sets.
The batch size is 32 training instances.
The stopping measures are evaluated every quarter epoch.
Figure A.12 shows the results averaged over 10 runs (the error is the standard deviation).
The difference between the top and the bottom panel is the data set and the patience parameters.
The x -axis depicts the fraction of the data set, which is warped for better accessibility.
In each panel, the left-hand side subplots depict the results of the reduced data set experiment where the right-hand side subplots depict the result of the permutation experiments.
The y -axis of the top subplot shows the accuracy on the non-reduced, non-permuted test set.
The y -axis of the bottom subplot shows when the stopping criterion was triggered.
We note the following observations, which hold for both panels: More, non-permuted data yields higher test accuracy.
Also, as expected, the optimal stopping gives the highest test accuracy.
The ﬁxed early stopping results in inferior test accuracy when only a fraction of the data is available.
The neural persistence based stopping is triggered late when only a fraction of the data is available which results in a slightly better test accuracy compared to training and validation loss.
The training loss stopping achieves similar test accuracies compared to the persistence based stopping (for all regimes except the very small data set) with shorter training, on average.
We note that, it is generally not advisable to use training loss as a measure for stopping because the stability of this criterion also depends on the batch size.
When only a fraction of the data is available, the validation loss based stopping stops on average after the same number of training epochs as the training loss, which results in inferior test accuracy because the network has seen in total fewer training samples.
Most strikingly, validation loss based stopping is is triggered later (sometimes never) when most training and validation labels are randomly permuted which results in overﬁtting and poor test accuracy.
To conclude, the neural persistence based stopping achieves good performance without being affected by the batch size and noisy labels.
The authors also note that the result is consistent for multiple architectures and most patience parameters.
[Figure]
T a b l e
A . 1 :
P a r a m e t e r s a nd hyp e r p a r a m e t e r s f o r t
h e e xp e r
i m e n t on b e s t p r ac ti
ce s a nd n e u r a l p e r
s i s t e n ce .
D r opo u t a nd b a t c h no r m a li za ti on w e r e a pp li e d a f t e r t
h
e
ﬁ r s t
h i dd e n l a y e r .
T h r oughou t t h e n e t w o r k s ,
R e L
U w a s t h e ac ti v a ti on f un c ti on o f c
ho
i
ce .
D a t a s e t # R u n s # E
po c h s
A r c h it ec t u r e
O p ti m i ze r
B a t c h S
i ze
H yp e r p a r a m e t e r s
M N I
S T  [; 650 ]
A d a m (cid:17) = 0 : (cid:12)= 0 :, (cid:12)= 0 : 999 , ϵ = 1 (cid:2) 10 (cid:0)(cid:17) = 0 :
(cid:12)= 0 :, (cid:12)= 0 : 999 , ϵ = 1 (cid:2) 10 (cid:0) 8 ,
B a t c h N o r m a li za ti on (cid:17) = 0 :
(cid:12)= 0 :, (cid:12)= 0 : 999 , ϵ = 1 (cid:2) 10 (cid:0) 8 ,
D r opou t%
T a b l e
A . 2 :
P a r a m e t e r s a nd hyp e r p a r a m e t e r s f o r t
h e e xp e r
i m e n t on ea r l y s t opp i ng .
T h r oughou t t h e n e t w o r k s ,
R e L
U w a s t h e ac ti v a ti on f un c ti on o f c
ho
i
ce .
D a t a s e t # R un s # E
po c h s
A r c h it ec t u r e
O p ti m i ze r
B a t c h S
i ze
H yp e r p a r a m e t e r s ( F a s h i on -) M
N I S
T
P e r ce p t r on
M i n i b a t c h S
GD (cid:17) = 0 :
A d a m (cid:17) = 0 : (cid:12)= 0 :, (cid:12)= 0 : 999 , ϵ = 1 (cid:2) 10 (cid:0)[0 ; 100 ] [ 20 ; 20 ; 20 ]
C
I F
A R -   [0 ; 300 ; 800 ]
A d a m (cid:17) = 0 : (cid:12)= 0 :, (cid:12)= 0 : 999 , ϵ = 1 (cid:2) 10 (cid:0)
I M
D B  [8 ;;]
A d a m (cid:17) = 1 (cid:2) 10 (cid:0)(cid:12)= 0 :, (cid:12)= 0 : 999 , ϵ = 1 (cid:2) 10 (cid:0)
[Figure]
A. T ESTING ACCURACY OF DIFFERENTLY REGULARIZED MODELS
We showed in the main text that neural persistence is capable of distinguishing between networks trained with/without batch normalization and/or dropout.
Figure A.13 additionally shows test set accuracies.