C ONVOLUTIONAL N EURAL N ETWORK D YNAMICS : A G RAPH P ERSPECTIVE
Anonymous authors
Paper under double-blind review
A BSTRACT
The success of neural networks (NNs) in a wide range of applications has led to increased interest in understanding the underlying learning dynamics of these models.
In this paper, we go beyond mere descriptions of the learning dynamics by taking a graph perspective and investigating the relationship between the graph structure of NNs and their performance.
Speciﬁcally, we propose (1) representing the neural network learning process as a time-evolving graph (i.e., a series of static graph snapshots over epochs), (2) capturing the structural changes of the NN during the training phase in a simple temporal summary, and (3) leveraging the structural summary to predict the accuracy of the underlying NN in a classiﬁcation or regression task.
For the dynamic graph representation of NNs, we explore structural representations for fully-connected and convolutional layers, which are key components of powerful NN models.
Our analysis shows that a simple summary of graph statistics, such as weighted degree and eigenvector centrality, over just a few epochs can be used to accurately predict the performance of NNs.
For example, a weighted degree-based summary of the time-evolving graph that is constructed based on 5 training epochs of the LeNet architecture achieves classiﬁcation accuracy of over 93%.
Our ﬁndings are consistent for different NN architectures, including LeNet, VGG, AlexNet, and ResNet.
I NTRODUCTION
Neural networks (NNs) have driven advancements in many domains, including computer vision and image processing (Hu et al., 2018), natural language processing (Sutskever et al., 2014;
Bahdanau et al., 2015), and bioinformatics (Cao et al., 2020; Li et al., 2019).
As task complexity increases, networks grow deeper and larger, consequently requiring more computational resources and training data, as well as sacriﬁcing interpretability for improved task performance.
Some works have focused on understanding and interpreting deep NNs (Raghu et al., 2017; Chakraborty & et al., 2017; Ioffe &
Szegedy, 2015; He et al., 2016).
One approach towards this goal involves representing the NN as its underlying graph structure, and studying selected graph properties, such as clustering coefﬁcient, path length
(You et al., 2020), modularity (Filan et al., 2021), persistence (Rieck et al., 2019).
For example, You et al. (2020) represent NNs as relational graphs capturing the message passing process, and investigate the correlation between the predictive performance of NNs and architectural changes.
However, the studies of NN structures as graphs are limited, and the structural changes of the underlying graph during the training process have been largely overlooked in the literature.
To ﬁll this gap, in this work, we take the graph perspective and aim to predict the performance of an NN by capturing early NN dynamics during the training phase.
Successful performance prediction based on only a few epochs could be used for early stopping (Yu & Zhu, 2020), and thus, more efﬁcient NN training.
To solve the performance prediction problem, we propose a multi-step framework, depicted in Fig.
Speciﬁcally, we propose to represent the underlying graph structure of an NN as a time-evolving k-partite graph, where each part corresponds to a different NN layer, and each graph snapshot in the evolving graph maps to an NN instance at a speciﬁc epoch.
We build on existing graph representations of fully-connected and convolutional layers, both of which are key components of popular NN architectures, and introduce a new, compact, efﬁcient-to-compute (“rolled”) graph representation for convolutional layers.
Then, we extract well-known node features (weighted degree and eigenvector centrality) from the time-evolving graph and construct temporal signatures by computing summary statistics on the node feature distributions.
Finally, we cast NN performance prediction as a classiﬁcation task and a regression task, each of which operates on the temporal structural signatures of the NN.
Our main contributions are:
• New graph-based NN representation: We introduce a new graph representation for convolutional layers that is more compact and efﬁcient-to-compute than the existing unrolled representation (Rieck et al., 2019), while not sacriﬁcing accuracy in the NN performance prediction task.
• Framework for NN performance prediction: We propose a simple, multi-step graph-based framework to solve the NN performance prediction problem by capturing the early NN dynamics during the training phase.
• Extensive empirical analysis: Using well-known image classiﬁcation datasets (ImageNet and CIFAR-10), and a variety of NN architectures (AlexNet, VGG, LeNet, and ResNet)
, we show that our framework can effectively predict the performance of NNs by observing only a few epochs of training, well before their corresponding early stopping epochs.
For instance, using our framework to capture the changes in the graph structure in only 5 training epochs of the ResNet architecture results in classiﬁcation accuracy of over 90%.
P RELIMINARIES
We ﬁrst present the key concepts that our work builds upon.
Table 1 gives the major symbols and their descriptions.
An NN model is a collection of connected units (neurons) that are organized in layers, and is deﬁned by a set of parameters that adjust during the training process.
We refer to the training process of a single architecture along with its respective hyperparameters as an ‘instance’.
We focus on two types of layers that are the key components of many powerful NN models such as LeNet(Lecun et al., 1998), VGG (Simonyan & Zisserman, 2015) and ResNet(He et al., 2016): fully connected layers (fc) and convolutional (conv) layers.
2. G RAPHS : T ERMINOLOGY AND N OTATION
Let G =
( V , E ) be an undirected, weighted graph with node set V , edge set E , and weighted adjacency matrix W ∈ R |V|×|V| .
The neighbors of node v are deﬁned as N G ( v ) =
{ u : ( u, v ) ∈ E} ; i.e., the set of all nodes that connect directly to v .
Graph G is k -partite if its nodeset V can be partitioned into k independent sets: V = (cid:83)
ki =1 V i and V i ∩ V j = ∅ , i (cid:54) =
j .
A time-evolving graph is a series of static graph snapshots over time: G = { G 1 , G 2 , ..., G T } .
The graphs in the series may have different nodesets and edgesets.
Here we focus on two of the most commonly-used node features in graph mining and network science, degree and eigenvector centrality, which capture different types of node importance or inﬂuence.
We discuss the interpretation of these features in the context of neural network dynamics in § 3.2.
• Degree : Degree is the simplest and most efﬁcient to compute node feature, and captures the connectivity of a node.
The weighted degree of node v is deﬁned as the sum of weights of the edges that are incident to i : d wi = (cid:80) j W ij .
Tracking changes in the degree of a node over time is the most direct way of capturing its structural evolution.
• Eigenvector centrality : This centrality is a sophisticated extension of degree centrality, related to Google’s PageRank, which indicates the inﬂuence of a node in G (Bonacich, 1972).
If a node is connected to several nodes with high eigenvector centrality, then that node will have high centrality.
The eigenvector centrality of node i is deﬁned as the i th element in the principal eigenvector v of W (i
[Table]
e., the eigenvector that corresponds to the largest eigenvalue λ ): Wv = λ max v .
2. F ULLY - CONNECTED LAYERS : G RAPH REPRESENTATION
[Table]
Let x ∈ R D in be the input vector
[Table]
W ∈ R D in × D out the learnable weight matrix, b ∈ R D out the bias vector and z a non-linear activation functions.
Then, the output of this layer is given as y =
z ( xW + b ) , where D in and D out correspond to the dimension of input and output layers.
Fully-connected layers are straightforward to represent with a weighted, undirected graph (Filan et al., 2021).
Each neuron, including those in the input and output layers, corresponds to a node.
Two neurons are connected via an edge if they appear in consecutive layers.
More formally, let nodeset V
i be the set of all neurons at layer i , and nodeset V j the set of all neurons at layer j (which follows layer i ).
The edges that connect all nodes across the two nodesets are deﬁned by the learnable weight matrix W ∈ R |V
i |×|V j | .
2. C ONVOLUTIONAL LAYERS : U NROLLED GRAPH REPRESENTATION
[Figure]
Let X ∈
R c × h × w be the input and
K ∈ R f
× c × h ker × w ker the convolutional kernel, where c is the number of channels, and h ker , w ker are the height and width of the kernel respectively.
After performing the convolution, we have Y ∈ R F × h out × w out , where h out = h − h ker + 1 and w out = w − w ker + 1 .
Typical multilayer CNNs consist of convolutional layers followed by fully-connected layers.
The representation of fully connected layers is straightforward but cannot be used to model convolutional layers.
Rieck et al. (2019) proposed to "unroll" the convolution to convert conv layers into graphs.
In Sec. 3, we introduce a new, compact, rolled representation that is both efﬁcient to compute and effective at predicting the NN performance, as we show empirically in Sec. 4.
For a convolutional layer, we ﬁrst unroll the convolutional operation and then represent the graph as in the case of a fully-connected layer.
In this representation, the nodes and edges of the graph are deﬁned through the convolutional operation as matrix multiplication (Gebhart et al., 2019).
Speciﬁcally, for an input image X and kernel K i ∈ R f i × c
i × h
i × w i , each node in layer l is deﬁned to be the output of the mapped feature of that input for each ﬁlter
f i .
Edges connect each of these nodes to the corresponding nodes of the output neuron in the next layer.
These edges are weighted by the activation value of that neuron (i.e., the input image for a speciﬁc stride is multiplied by the ﬁlter value at that location in the image).
For an input image ﬁlter size ( c i × h
i × w i ) that results in feature-map (output) of size 1 × o h × o w , the number of nodes of output layer of the graph representation is a function of o h × o w , while the number of nodes in the conv operation layer of graph is a function of c i × h
i × w i .
As an example, Fig. 2 depicts the unrolled graph representation of a toy convolutional layer.
It shows one conv operation on a small 2-dimensional input image ( 4 × 4 ) and ﬁlter ( 2 × 2 ).
The nodes and edges of the represented graph are shown based on one stride (of size 2) of convolution operation.
N EURAL N ETWORK P ERFORMANCE P REDICTION : A T EMPORAL G RAPH - BASED A PPROACH
In this section, we ﬁrst formally introduce the problem that we seek to solve.
Then, we present a new, compact, and efﬁcient-to-compute graph representation for convolutional layers (§ 3.1), and describe our proposed temporal graph-based framework that captures the NN dynamics during training (§ 3.2).
Problem 1 (NN Performance Prediction) Let N = { N tr 1 , N tr 2 , . .
, N tr n } be a training set of n NNs trained for T epochs and A = { α 1 , α 2 , . . .
, α n } their corresponding downstream task accuracies (e.g., for image classiﬁcation).
We seek to predict the accuracy α tst of a new instance N tst trained for a very small number of t (cid:28) T epochs by using t epochs for the trained NNs in N .
Our proposed solution takes the graph perspective, and its ﬁrst step is to represent each NN as a temporal graph.
In addition to the graph representations that we presented in Sec. 2, we introduce a new, efﬁcient representation for convolutional layers, which we describe next.
3. R OLLED G RAPH R EPRESENTATION FOR C ONVOLUTIONAL L AYERS
We call our proposed graph representation for conv layers “rolled,” since it avoids unrolling the convolutional operations introduced in (Rieck et al., 2019).
Overview & Motivation.
To preserve the semantic meaning of conv layers, we represent each ﬁlter as a node , and link ﬁlters in consecutive layers via weighted edges (we deﬁne the weights below), as shown in Fig.
The motivation behind this approach is that for larger networks, there is an explosion of nodes by unrolling the convolutions (unrolled representation explained in § 2.3), and the integrity of a unit (a single ﬁlter or kernel) becomes untenable.
Also, by mapping the nodes to speciﬁc entities of NNs, our proposed graph model is more interpretable than the unrolled model, and thus it is easier to interpret the outputs of downstream graph analysis on our graph representation (e.g., computing node features, tracking the evolution of the graph).
Detailed Description.
Formally, let tensor K i be a kernel in layer i with f i ﬁlters, each with c i channels and dimensions h
i × w i .
We use bracket notation to index into the kernel: for example, K i [ l, : , : , :] indexes the l th ﬁlter of kernel K i .
We create f i nodes representing each ﬁlter { v ( i ) 1 , v ( i ) 2 , ...v ( i ) f i } , with features deﬁned as the corresponding biases in that layer.
Let K j be the next convolutional layer deﬁned analogously.
While edge weights between neurons in FC layers are deﬁned in the standard way, we deﬁne the edge weights between conv layers as the norm over each kernel’s channels.
The edge between node v ( i ) k representing the k th ﬁlter in layer i (i.e., K i [ k, : , : , :] ) and node/ﬁlter v ( j ) l in layer j
(i.e. K j [ l, : , : , :] ) has weight w v ( i ) k ,v ( j ) l = norm ( K j [ l, k, : , :]) , which is the norm of the k th channel of the l th ﬁlter in the j th layer.
Though the proposed representations enable alternative edge weight conﬁgurations, we focus on the norm of ﬁlters as other studies, including those on pruning NNs (Li et al., 2016), have demonstrated that this metric has strong correlation with ﬁlter importance.
In the case of two conv layers, the resultant graph is an attributed bipartite graph with f i + f j nodes and f i × f j edges, where node attributes include ﬂattened weight vectors or ﬁlter maps.
Other information such as average gradients or the bias vector can also be used for node features.
3. P ROPOSED T EMPORAL G RAPH - BASED F RAMEWORK
A key objective of our work is to test whether the introduced graph representation of NNs is informative for predicting the performance of NNs.
Next, we describe the steps of our proposed framework for solving Problem 1, namely performance prediction from the NN training dynamics.
As shown in Fig.
1, our method consists of four steps: (S1) generation of a temporal graph for the training phase of each NN; (S2) extraction of node features; (S3) construction of a feature-based graph signature that captures the NN dynamics; (S4) prediction of NN performance by training a classiﬁer or regressor on the constructed signatures.
We describe these steps in more detail next.
(S1) Graph generation.
The ﬁrst step involves converting the training process of each input NN into a time-evolving graph.
Each NN N tr i ∈ N , is represented as a series of checkpoints saved for t training epochs.
Based on these checkpoints and the graph representation approaches for fc and conv layers presented in 2.2, 3.1 and 2.3, we ﬁrst convert the τ th NN checkpoint into weighted graph G ( τ ) i at timestamp/epoch τ .
We note that although our proposed graph representation involves node features, our framework does not leverage them, and thus we consider the generated graphs unattributed.
Therefore, each NN N tr i is mapped to a time-evolving graph G tr i = { G 1 i , G 2 i , ..., G ti } .
The output of this step is a set of n time-evolving graphs {G tr 1 , G tr 2 , ..., G tr n } corresponding to the original n NNs in N .
(S2) Feature extraction.
Next, the goal is to capture the structural dynamics of the NN training process.
We aim to select graph measures that can capture changes during the training process, take into account the edge weight of graphs and can be calculated efﬁciently.
In order to do that in an interpretable way, we extract two well-known node centralities from each snapshot of each generated time-evolving graph G i : weighted degree centrality and eigenvector centrality (§ 2.1).
The weighted degree is a simple function of the learnable weight matrix W during the training phase of NN, therefore it gives us insights into the training dynamics at the node/neuron/ﬁlter level.
The eigenvector centrality is an extension of the degree centrality, which captures the highly inﬂuential nodes, and has been successfully used in neuroscience to capture the dynamic changes of real neural networks (or connectomes) (Lohmann et al., 2010).
Eigenvector centrality can be used to capture importance and connectivity of ﬁlters/neurons (i.e., the nodes in our graph representation).
Also, eigenvector centrality has been used for detecting communities (Newman, 2006) or clusters (Wu et al., 2013), and thus provides structural information about the clusterability of the NN, which is complementary to that provided by the simpler and more efﬁcient-to-compute degree centrality.
Our choice of features is also guided by the inherent k -partite structure of our proposed graph representation, which cannot be represented well by several other commonly-used graph features.
For example, clustering-based features (e.g., number of triangles, transitivity, clustering coefﬁcient) and cycle-based metrics which account for closed paths in a graph are always equal to 0 for k -partite graphs.
Moreover, connected component-related features (i.e., strong/weak connectivity) do not capture the learned edge weights, which are important for modeling the training dynamics of NNs.
Other features (e.g., betweenness centrality) tend to be computationally expensive, and would add signiﬁcant overhead compared to early stopping methods.
(S3)
Graph signature construction. In order to be able to compare NN-based graphs (with different number of nodes and edges), we summarize the structural changes in the generated time-evolving graphs at the graph level (rather than the node level, as in (S2)), and construct a statistical summary of the extracted node centralities (signature) per time-evolving graph G i .
For each snapshot G ( τ ) i of G i , we create a signature vector using ﬁve node feature aggregators, which were introduced in (Berlingerio et al., 2012) for graph similarity: median, mean, standard deviation, skewness, and kurtosis , where all but the median are moments of the corresponding distribution.
Thus, G ( τ ) i is mapped to a (static) signature vector s ( τ ) i ∈
R 5 , representing the statistical summary of its node features (i.e., degree or eigenvector centrality) at time τ .
To put more emphasis on the most recent timestamp, we can redeﬁne the signature at time τ as the linear weighted average of the signatures up to that point, s ( τ ) i ←
[Equation]
, or an exponential function of the previous signatures, s ( τ )
i [Equation]
To obtain the temporal signature of the evolving graph G i , we aggregate the (static) signatures up to timestamp/epoch t : s ti =
s 1 i ⊕ s 2 i ⊕ ...
⊕ s ti , where ⊕ denotes concatenation.
We note that global features such as algebraic connectivity, modularity, and average shortest paths may be seen as alternative ways for constructing global graph signatures while circumventing the local feature extraction step (S2); however, these features fail to capture the structural changes in our proposed graph representations (they remain (near-)constant over time) and lead to poor performance.
(S4)
Performance prediction.
For the last step of performance prediction, we consider two tasks:
[Table] • Classiﬁcation: We train a classiﬁer (e.g., SVM, MLP) using the training graphs {G tr 1 , G tr 2 , ..., G tr n } represented by their temporal signatures { s ttr 1 , s ttr 2 , ..., s ttr n } , and their corresponding accuracies A
= { α 1 , α 2 , . . .
, α n } mapped to labels L = { l 1 , l 2 , . . .
, l n } (e.g., high/low accuracy) based on some threshold.
Any test NN instance is then classiﬁed using the trained classiﬁer.
• Linear regression: We perform linear regression to estimate the actual accuracy value α tst of a new test instance N tst based on its signature obtained through steps ( S1 )-( S3 ).
E MPIRICAL A NALYSIS
In this section
[Table]
we empirically evaluate the effectiveness and efﬁciency of our framework in the classiﬁcation and regression tasks for different graph representations (rolled and unrolled graphs for conv layers) and different feature-based signatures (degree- vs. eigenvector centrality-based).
Data.
We investigate NN dynamics using our framework on two well-known image classiﬁcation datasets CIFAR-10 (Krizhevsky, 2009) and ImageNet (Russakovsky et al., 2015)
CIFAR -10 consists of 50K training images and 10K test images.
For ImageNet, we use a sample that has 50K training images and 5K validation images used as the test set.
The sample is obtained by randomly selecting 100 classes from Tiny ImageNet (tin) and downsizing the images to 32 × 32 colored images.
Conﬁguration.
The conﬁguration of training different NN models, and the early stopping method are described in App. A.1.
For the unrolled graph representation, which is signed, we consider different graph types (e.g., positive, negative), which we describe in App.
A.2.
4. C LASSIFICATION : P REDICTING NN ACCURACY RANGE FROM NN TRAINING DYNAMICS
Task setup.
We cast the NN performance prediction as a classiﬁcation task.
Speciﬁcally, the generated time-evolving graphs are labeled as high and low accuracy based on the performance of their corresponding NNs; Table 2 lists the threshold value chosen for low and high accuracy labels based on the ﬁnal accuracy range of trained NNs, as well as the early stopping epochs for each architecture.
Five-fold cross validation is used to predict the label of the test graphs in a binary classiﬁcation task using SVM and MLP, where the input is the set of temporal signatures { s ttr 1 , s ttr 2 , ..., s ttr n } .
We report the classiﬁcation accuracy.
Since the sample of NNs (App. A.1) is randomly selected with balanced high/low accuracy instances, the accuracy of a random classiﬁer as the baseline is 50% (omitted from the charts to avoid clutter).
Additionally, to show that our proposed graph representation and signatures are general and can be useful across different NN architectures, we consider the following setup: we train the classiﬁer on a small set of NN models (i.e., different architectures—such as LeNet, AlexNet, and VGG— and hyperparameters), and predict the performance on unseen architectures (e.g., ResNet).
We describe these experiments in App.
A.4.
Results.
Figure 4 illustrates the performance of SVM and MLP classiﬁers operating on weighted degree -based signatures of the rolled and unrolled graph representations of the LeNet and AlexNet architectures, trained on the CIFAR-10 dataset.
We omit the results on VGG and ResNet, as well as image classiﬁcation on ImageNet, because the unrolled graph generation process is prohibitively expensive, both in terms of time and space.
Overall, the rolled and unrolled graph representations show similar trends in classifying NNs by effectively capturing their early training dynamics: the structural changes in the training NN architectures during the ﬁrst 6-15 iterations are sufﬁcient to classify the performance of NN instances with over 90% accuracy.
However, as we discuss in § 4.3, our proposed rolled representation is signiﬁcantly more efﬁcient than the unrolled representation, and can generalize to deeper and larger NNs.
We provide more details for these experiments in App.
A.2.
[Figure]
O BSERVATION
Both the rolled and unrolled time-evolving graph representations of NNs are effective in capturing the changes in the NN dynamics during the training phase, and can be used to predict the accuracy of an NN instance after observing only a few training epochs.
Our proposed rolled representation is also space- and time-efﬁcient, unlike the unrolled representation.
In the remainder of this analysis, we focus on the rolled representation, which is more efﬁcient for larger NN models and datasets.
We present the classiﬁcation results for both types of signatures for the temporal graphs corresponding to the training dynamics of VGG and ResNet-44 (CIFARdataset) in Fig.
4(e)-(h) and LeNet, AlexNet and ResNet-50 (ImageNet datatset) in Fig.
In addition to the results discussed above for LeNet and AlexNet, we provide the NN classiﬁcation accuracy for the eigenvector-based signatures in Fig.
11 in the appendix.
In all the cases, classiﬁcation accuracy of 80-95% is achieved in less than 10 training epochs.
For degree-based signatures, SVM tends to outperform MLP, while the trend is reversed for eigenvector-based signatures.
For example, for both
VGG and AlexNet for the CIFAR-10 image classiﬁcation task, MLP can predict the performance with accuracy ∼ 95% using the eigenvector-based signatures from the ﬁrst 6 training epochs; the same trend is observed on ImageNet for the LeNet and AlexNet architectures.
For ResNet on the ImageNet classiﬁcation task, SVM tends to perform well for both degree and eigenvector centrality signature vectors; while MLP tends to perform poorly in this case, the MLP variant operating on the exponential average of signature vectors outperforms the original MLP and all SVM variants.
In all the cases, both classiﬁers reach performance over 80%-90% signiﬁcantly before the early stopping point for all the architectures.
In general, for ImageNet, we observe that the signatures based on eigenvector centrality are more effective in the NN performance prediction task compared to the weighted degree-based signatures—irrespective of NN architecture.
O BSERVATION
For the CIFAR-10 image classiﬁcation task, the rolled graph representation for all
NN architectures and both signature types achieve accuracy of > 90%.
For ImageNet, the eigenvector centrality-based signatures tend to yield higher performance compared to the weighted degree-based signatures, though both achieve accuracy of 80-90%.
4. R EGRESSION A NALYSIS : P REDICTING NN ACCURACY FROM NN TRAINING DYNAMICS
In this section, we discuss the results and key ﬁndings of predicting the actual performance (accuracy) of an NN using a linear regression model, instead of treating this task as a classiﬁcation problem as in the previous section. [Table] [Figure]
[Table]
[Figure]
Task setup.
Following the same experimental methodology as in the classiﬁcation task, we use the degree-based signatures of the generated time-evolving graph as the predictor and the overall accuracy of the corresponding NN as the dependent variable.
We use 20% of our observations as a test set, and the rest as a training set.
We report the testing mean absolute error (MAE) and coefﬁcient of determination ( R 2 ).
Table 3 displays the results for regression prediction on ImageNet dataset, the prediction model show similar trend as CIFAR-10 on this datatset.
The MAE values are smaller for this dataset since the accuracy range of both architectures are much narrower than the CIFAR-10 dataset (See Table 2 for accuracy ranges of different architectures and datasets.)
Results.
Table 4 summarizes the regression results for four architectures where the independent variable is degree features of rolled graph representation.
For each step of this experiment, we use a concatenation of two consecutive feature vectors as independent variable of regression model.
For LeNet, at timestamp 5-7 the regression model error (MAE for test set) is 5.74, while the accuracy range of LeNet for this dataset is wide (0.9-73.8), and based on R 2 = 0 .
91 , we can interpret that 91% of variant in accuracy of NN can be explained by the degree changes of the rolled graph between timestamp 5-7.
A similar trend is observed for AlexNet, VGG and ResNet, where after observing a few epochs, the prediction model shows a very low MAE with high R 2 .
O BSERVATION
In sum, the weighted degree-based signature vector of time-evolving graphs generated based on the rolled graph representation is a strong predictor of the actual accuracy value of NNs.
For both CIFAR-10 and ImageNet, we show that, by observing only a subset of early training epochs, we can effectively predict the accuracy value of NNs with a small MAE and a high coefﬁcient of determination R 2 > 0 .
4. D ISCUSSION
[Table]
Time efﬁciency and early stopping.
Figure 6 represents the total average runtime of NN training for early stopping of each architecture (green bars) with comparison of average run-time of graph generation, degree calculation and eigenvector centrality calculation for the number of epochs that were needed for that architecture to achieve the highest accuracy in classiﬁcation task.
For all the 3 architectures [Figure] rolled graph representation can achieve a high accuracy prediction much faster than the early stopping of NN training.
But for unrolled representation, this is only true for LeNet architecture.
As the size of NN increases, the unrolled graph generation gets slower and training with early stopping method is faster than our prediction framework.
Size analysis.
Table 5 shows the number of nodes and edges of the rolled graph representations compared with the unrolled representation, our proposed rolled representation of NN reduces the complexity of NN architectures in terms of #nodes and #edges in the graph while retaining comparable performance on the prediction tasks.
Our proposed rolled graph representation has big advantage in terms of size over the unrolled graph representation, especially for deeper networks.
Limitations.
The unrolled representation is not scalable therefore cannot be applicable for larger networks and datasets.
The main advantage of this representation is that there is no loss of information during the training phase of NNs, and we can interpret the learning process easier than the rolled method.
For example, we showed that the negative subgraph alone would be a stronger predictor of the accuracy of NNs, which can lead us to another direction of signed graph analysis in further study of the learning process of NNs.
Complexity analysis of the rolled graph representation.
For an NN with n f total number of ﬁlters in all the conv layers and n n total number of neurons in the fully connected layers, modeling the nodes takes a constant time O ( |V| ) , where V is the set of generated nodes in the graph and |V| < = n f + n n (due to dropout some nodes are removed).
The computation of the edge weights is
O ( (cid:80) l |V| l ∗ |V| l +1 ) , where |V| l are the nodes at layer l .
R ELATED W ORK
We cover the most relevant work here, and dynamic graph mining in App. A.5.
Some recent research efforts are devoted to modeling the NN architectures as graphs due to their topological identity and study their graph properties Rieck et al.
; Filan et al.
; You et al.
For example, You et al.
You et al.
(2020) propose a relational graph representation to model message exchange between layers, and empirically show the common properties shared by NNs with signiﬁcantly improved predictive performance in terms of graph clustering coefﬁcients and average path lengths.
Rieck et al. Rieck et al.
(2019) propose a complexity measure related to NN performance—neural persistence—based on topological data analysis on weighted stratiﬁed graph.
Filan et al. Filan et al.
(2020) present an exploratory study of the NN modularity.
Gebhart et al.
(2019) proposes to compute persistent homology over the activation graph of an NN.
The output is a graded set of subgraphs, which are shown to be related to the task-speciﬁc semantic that are captured by original NN.
C ONCLUSION
In this work, we investigated the early training dynamics of NNs from a time-evolving graph perspective.
We proposed a new graph representation to efﬁciently convert convolutional layers into compact and intuitive graph structures.
Then, we showed that a simple, temporal graph signature based on summary statistics of the degree or eigenvector centrality distributions over only a few epochs can be used as a strong predictor variable to estimate the accuracy of NNs in downstream tasks (e.g., image classiﬁcation).
Exploring the role of our efﬁcient proposed framework for early stopping is a promising future direction.
R EPRODUCIBILITY S TATEMENT
For reproducibility we provide the references to the datasets and existing (rolled) graph representation of convolutional layers in § 4.
In App.
§ A. we provide the detailed hyperparameter settings for NN training.
We will also make our code publicly available upon acceptance
A A PPENDIX A. E XPERIMENTAL S ETUP : C ONFIGURATION
We ﬁrst train ﬁve different NN architectures, LeNet-5 (Lecun et al., 1998), VGG13 (Simonyan &
Zisserman, 2015), AlexNet (Krizhevsky et al., 2012), ResNet-32 and ResNet-44 (He et al., 2016) , on
CIFAR-10 dataset and three architectures, LeNet-5, AlexNet and ResNet-50, on ImageNet.
During training, the NN parameters are updated using stochastic gradient descent.
On each dataset, we train NNs by combining 48 learning rates { 1 , 1 . 5 , . . .
, 4 . 5 } × { 10 − 6 , . .
, 10 − 1 } , and 10 dropout rates { 0 , 0 . 1 , . . .
All the models are trained for 50 epochs with batch size 128 and an early stopping method by which the training stops when the testing accuracy does not increase forconsecutive epochs.
Consequently, we obtain 480 NNs per architecture and dataset, with diverse performance.
Information for the generated NNs is summarized in Table 2: the range of epochs at which the training stops early, the range of ﬁnal testing accuracy for the trained NNs, and the accuracy threshold used to map actual NN performance to ‘low/high accuracy’ labels for the classiﬁcation task.
We trained the NNs on an Nvidia 1080Ti GPU with 11G memory, and we conducted all the other experiments on 2.60GHz Intel Xeon E5-2697 v3 platform with 1024G memory.
For computational efﬁciency and to avoid having a largely imbalanced dataset, out of the 480 NNs, we randomly sampled 250 NNs with an even split of high- and low-accuracy networks.
For this sample, the ﬁrst t epochs of training for each conﬁguration were saved as checkpoints to be converted to time-evolving graphs (step (S1) ).
A. U NROLLED G RAPH R EPRESENTATION
In these experiments, we split the represented graph into two subgraphs with positive and negative edge weight 1 .
Then we used the two graphs G pos and G neg to calculate the degree and eigenvector centrality feature vector summary for both of them, and concatenate those feature vectors to represent the whole graph G pos − neg .
According to Figure 4, weighted degree signature vector of G neg is a good descriptive variable to predict performance of NNs.
For LeNet, we achieve an accuracy of 90% with only 3-5 epochs.
For AlexNet, since the graph generation process was slow, we constructed the time-evolving graph based on a few sub-samples of training epochs where
T = 1 , 5 , 10 , 15 .
The results show that even with a fewer sequence of observations the classiﬁer can achieve a high value of accuracy.
For instance, at timestamp 10, in which the feature vector is the concatenation of degree features at epochs 1, 5, and 10, the accuracy of the classiﬁer is more than 85%.
Based on results in Figure 11a, signature vector calculated based on eigenvector centrality of G neg is also very good descriptive variable to predict performance of NNs and can achieve accuracy of 90% with only 3-5 epochs for LeNet.
For all the experiments on unrolled representation, the feature vector only based on G neg achieves very high accuracy similar to or higher than that of G pos − neg , while feature vectors only based on
G pos result in the poorest accuracy among them.
A. F EATURE A NALYSIS
Classiﬁcation feature analysis.
Figure 7a shows the weight of each statistical aggregator for graph degree of VGG architecture for CIFAR-10.
The top three highly weighted features are mean, standard deviation and median of degree.
Figures 7b, 7c and 7d represent the change in value of those features respectively over time for high and low accuracy instances.
We observe that the statistical aggregator of degree for high accuracy graphs tends to increase over time while for most of low accuracy one these values do not change over time or the change is very small.
Figure 8 depicts similar trend for the
LeNet architecture, while the values of degree mean and standard deviation for low accuracy cases tend to have a very big spike in the ﬁrst 3 epochs of training but after that they follow a ﬂat line trend and tend to not change.
Figure 9 illustrates similar trend for ResNet-44 architecture and CIFARdataset.
The mode degree of graphs based on high accuracy cases tends to change drastically over time while the low accuracy cases do not change and show a ﬂat line pattern.
[Figure]
[Figure]
[Figure]
[Figure]
A. G ENERALIZING TO U NSEEN A RCHITECTURES
In this section, we broaden the scope of our empirical setup to show that our proposed graph representation and signatures are general and can be useful for performance prediction across different
NN architectures.
In this setup, we fully train just a small set of different NN architectures (i.e., different architectures and hyperparameters), and predict the performance on unseen NN architectures.
We consider two sets of experiments:
[Figure]
We train our classiﬁer on the smaller ResNet architectures (ResNet-32 for CIFAR-10, and ResNet34 for ImageNet), and test the performance of the larger ResNet architectures (ResNet-44 for CIFAR-10, and ResNet-50 for ImageNet).
In Figs.
10a and 10b, we observe that our proposed framework is able to accurately predict the performance level of the previously unseen, large ResNet architectures.
Our results show that the proposed temporal signatures can be used in a generalized scenario to predict the accuracy level of the same architecture with different numbers of layers (on the same dataset).
This generalization from small to bigger architectures is important since it is faster to train the smaller architectures.
In Figs. 10c and 10d, we see that our proposed method also successfully predicts the performance level of a new architecture (i
e. ResNet) when the training set is a combination of older architectures (LeNet, VGG, AlexNet).
For both of the experiments, we set a universal threshold value to label graphs in the train and test set.
For the training set of classiﬁers, we randomly choose a subset of the NNs with balanced high/low accuracy labels.
The size of the training set in the ﬁrst experiment is 250 and in the second experiment is 400.
The experiments were repeated 5 times and the average accuracy of classiﬁcation is reported.
A. A DDITIONAL R ELATED W ORK
Dynamic Graph Mining.
Dynamic graphs are mostly modeled as a sequence of edge additions and/or edge deletions.
To mine the dynamic graphs, traditional approaches leverage graph properties such as node centralities or motifs Riondato et al.
(2017); Paranjape et al.
(2017); Kovanen et al.
Despite their simplicity, these approaches show effectiveness in temporal tasks such as event detection Paranjape et al.
and structural prediction Aggarwal & Subbian (2014).
Recent embedding-based approaches tend to model dynamic graphs as a sequence of discrete-time snapshots and simultaneously represent the graph structure of each snapshot as well as the temporal evolution using deep neural networks such as GRU/LSTM Sankar et al.
(2020); Singer et al.
; Pareja et al.
DySAT Sankar et al.
(2020) leverages self-attention to compute node representations by jointly modeling graph structural property and temporal dynamics.
EvolveGCN Pareja et al.
(2020) uses GCN to generate node embeddings for the past snapshots, and learns parameters of the hidden layer for the next snapshot using GRU/LSTM.
Unlike these methods, we propose two different approaches to represent the underlying graph structure of NN, and use a very simple and efﬁcient dynamic graph signature feature vector to predict the accuracy of corresponding NNs.
A. A DDITIONAL F IGURES AND T ABLES
[Figure]
[Figure]
[Table]