E MPIRICAL S TUDIES ON THE C ONVERGENCE OF F EATURE S PACES IN D EEP L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
While deep learning is effective to learn features/representations from data, the distributions of samples in feature spaces learned by various architectures for different training tasks (e.g., latent layers of AEs and feature vectors in CNN classiﬁers) have not been well-studied or compared.
We hypothesize that the feature spaces of networks trained by various architectures (AEs or CNNs) and tasks (supervised, unsupervised, or self-supervised learning) share some common subspaces, no matter what types of DNN architectures or whether the labels have been used in feature learning.
To test our hypothesis, through Singular Value Decomposition (SVD) of feature vectors, we demonstrate that one could linearly project the feature vectors of the same group of samples to a similar distribution, where the distribution is represented as the top left singular vector (i.e., principal subspace of feature vectors), namely P -vector.
We further assess the convergence of feature space learning using angles between P -vectors obtained from the welltrained model and its checkpoint per epoch during the learning procedure, where a quasi-monotonic trend of convergence to small angles has been observed.
Finally, we carry out case studies to connect P -vectors to the data distribution, and generalization performance.
Extensive experiments with practically-used MLP, AE and
CNN architectures for classiﬁcation, image reconstruction, and self-supervised learning tasks on MNIST, CIFAR-10 and CIFAR-100 datasets have been done to support our claims with solid evidences.
I NTRODUCTION
Blessed by the capacities of feature learning, deep neural networks (LeCun et al., 2015) have been widely used to perform learning tasks, ranging from classiﬁcation, to generation (Goodfellow et al., 2014; Radford et al., 2015), in various settings (e.g., supervised, unsupervised, and self-supervised learning).
To better euclid the features learned by deep models, numerous works have studied on interpreting the features spaces of the well-trained models (Simonyan et al., 2013; White, 2016; Zhu et al., 2016;
Bau et al., 2017; 2019; Jahanian et al., 2020; Zhang & Wu, 2020).
Invariance beyond the use of architectures and labels.
While existing studies primarily focus on the interpolation of a given model to discover mappings from the feature space to outputs of the model (e.g., classiﬁcation (Bau et al., 2017) and generation (Jahanian et al., 2020)), the work is so few that compares the feature spaces learned by deep models of varying architectures (e.g.,
MLP/CNN classiﬁers versus Autoencoders) for different learning paradigms (Chen et al., 2020;
Khosla et al., 2020; Spinner et al., 2018).
More speciﬁcally, we are particularly interested in whether there exists certain “statistical invariance” in the feature space, no matter what type of architectures or whether label information (e.g., supervised vs. unsupervised vs. self-supervised (Chen et al., 2020) learning) are used in feature learning with the same training dataset.
Hypotheses.
It is not difﬁcult to imagine that the feature spaces of well-trained DNN classiﬁers in supervised learning setting might share some linear subspace (Vaswani et al., 2018).
When models are well ﬁtted to the same training set, the feature vectors of training samples should be projected to the ground-truth labels after a Fully-Connected Layer (i.e., a linear transform), while such linear subspace are supposed to distribute samples in a discriminative manner.
We doubt that such subspace might be not only shared by supervised learners but also with AEs which are trained to
[Table] [Table]
[Table] [Table]
d
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure]
[Table]
[Figure]
present cosine (in the range of [0,1]) of angles between the P -vectors of well-trained models of various architectures under different learning paradigms, using training and testing datasets respectively.
A well-trained model here is the one trained under the suggested settings after 200 epochs for supervised/self-supervised CNN classiﬁers and 100 epochs for unsupervised AEs.
Figure 1.
(c)–(h) present angles of P -vectors between the well-trained model and its checkpoint per training epoch of three learning paradigms, where the trends of convergence has been observed in all models, no matter whether the feature extractors of these models are trained with / without labels.
More discussion are provided in Section 4.
reconstruct input data without any label information in an unsupervised manner, or even shared with self-supervised DNN classiﬁers (e.g., SimCLR (Chen et al., 2020)) which train (1) CNN feature extractor (using contrastive loss without labels) and (2) linear classiﬁers (using discriminative loss based on labels) separately in an ad-hoc manner.
More speciﬁcally, we hypothesize that (H.I:) there exists certain common feature subspaces shared by well-trained deep models using the same training dataset, even though the architectures (MLPs, CNNs, and AEs) and the learning paradigms (supervised, unsupervised, and self-supervised) are signiﬁcantly different.
Further, as the training procedure usually initializes the DNN models from random weights and learns features from the training set step-by-step, we hypothesize that (H.II:) the training procedure gradually shapes the feature subspace over training iterations and asymptotically converge to the common subspace in certain statistical measure.
Finally, we hypothesize that (H.III) the convergence to the common feature subspace would connect to the data distribution and performance of models, as such behavior indicates how well the features are learned from data.
Contributions.
To test above three hypotheses, this work makes contributions in proposing new measures to the DNN features, namely P -vectors, and conducting extensive experiments for empirical studies.
We train deep models using various DNN architectures, multiple learning paradigms, and datasets, with the checkpoint restored per epoch.
Then, we extract the feature vectors for either training or testing sample sets, from the model (Please see Section 3 for details) and discover some interesting relationships or associations as discussed below.
I. P -vector and Convergence:
Given the matrix of feature vectors (#samples 1 × #features) for either training or testing samples, we perform the singular value decomposition (SVD) to obtain left and right singular vectors, characterizing the subspaces that samples distribute and the projection of features to subspaces respectively.
We observe that deep models well-trained using the same dataset share similar top left-singular vectors (referring to the principal subspace of feature vectors), namely P -vector in this study.
For example, Figure 1 (a)–(b).
show the high cosine similarity (close to 1) between the P -vectors of any two well-trained DNNs for eight different deep learning architectures/tasks.
This observation well backups H.I, where the common subspace in feature spaces shared by models trained with different architectures/paradigms has been observed.
With checkpoints obtained along the training procedure, we estimate the angle that compares P -vectors of the checkpoint per epoch and the well-trained one.
We ﬁnd such angle decrease over the number of epochs (in an overall manner) and would converge to a small one.
For example, Figure 1 (c)–(h).
demonstrate the consistent converging trends of angles between P -vectors of the well-trained model and its checkpoints in progress of training for eight different deep learning architectures/tasks.
This observation supports H.II, where the convergence to the common feature subspace has been observed.
II.
Data Distribution and Performance:
To connect to the data distributions, we intend to compare feature vectors and original data vectors of samples using P -vectors.
In addition to the P -vector estimated from the feature vectors of a deep model (namely a “model P -vector”), We simply form the sample vectors (either for training or testing) into a #samples × #data dimension matrix and perform
SVD to obtain the top left singular vector as the P -vector of samples (namely a “data P -vector”).
We estimate the angles between the model and data P -vectors, and ﬁnd a trend of convergence (from nearly orthogonal to relatively small angles), where we can see the well-trained models would incorporate smaller angles than ones in the early stage of training processes.
We further correlate such angles with training and testing accuracy of the models, where we observe signiﬁcant negative correlations in most cases of experiments.
The trends show that the model with a smaller angle between the model and data P -vectors would enjoy better performance.
The evidences backup H.III.
R ELATED W ORK
In this section, we ﬁrst present the preliminaries in understanding feature learning of DNNs, then discuss the most relevant works to our studies.
As early as 2013, (Simonyan et al., 2013) proposed to visualize the features learned by deep convolutional neural networks (CNN) and made sense of discriminative learning via deep feature extraction.
For generative models, (White, 2016; Zhu et al., 2016) studied the interpolation of latent spaces while (Zhu et al., 2016) discovered an user-controlled way to manipulate the images generated through the surrogation of latent spaces via manifolds.
Later, (Bau et al., 2017) presented the visual concepts learned in the feature spaces of discriminative models through network dissection on speciﬁc datasets while the same group of researchers also proposed GAN dissection (Bau et al., 2019) – an interactive way to manipulate the semantics and style of image synthesis.
(Richardson & Weiss, 2018) compared GAN and Gaussian Mixture Models (GMMs) to understand the capacity of distribution learning in GAN.
(Berthelot et al., 2019) proposed to improve understanding and interpolation of Autoencoders using adversarial regularizer while (Spinner et al., 2018) compared
AEs with its variational derivatives to interpret the latent spaces.
More recently, (Saxe et al., 2019) mathematically analyzed the process of neural representation construction from the perspectives of learning dynamics of deep neural networks.
(Jahanian et al., 2020) studied the “steerability” of
GAN, where it discovered point-to-point editing paths for content/style manipulation.
(Zhang &
Wu, 2020) uncovered the phenomena that DNN classiﬁers with piecewise linear activation tend to map the input data to linear subregions.
Other impressive studies in this line of research include (Nguyen et al., 2016; Arvanitidis et al., 2018; Sercu et al., 2019).
Discussion.
The most relevant studies to our work are (Bau et al., 2017; Zhang & Wu, 2020; Saxe et al., 2019; Lee et al., 2019).
For discriminative models, (Bau et al., 2017) recovered visual features learned by CNN classiﬁers with a priorly labeled dataset, and quantiﬁed then compared the feature learning capacities (namely “interpretability” in the work) of different DNN models through patterns matching with the ground truth.
Compared to (Bau et al., 2017), we carry out the empirical studies on a wide range of datasets without any prior information on their features and observe consistent phenomena in the distribution of samples in the feature spaces.
Furthermore, while (Zhang & Wu, 2020) studied the properties of regions where a supervised DNN classiﬁer with piecewise linear activation behaves linearly, our work observes the common linear subspaces shared by the features learned by the networks that are trained with different architectures (e.g., MLP/CNN classiﬁers and AEs with
ReLU activation) and paradigms (e.g., supervised, unsupervised, and self-supervised learning).
Furthermore, both our work and (Saxe et al., 2019) propose to compare feature/representation learning by various models through SVD, while we perform SVD to investigate the distribution of samples in feature space and (Saxe et al., 2019) uncovered the latent structures in input-and-output of neurons.
To understand the latent space of generative models, some statistical/mathematical tools (Richardson & Weiss, 2018; Lee et al., 2019) have been introduced.
Through comparisons between GAN and
# s a m p l e s 𝑛 #features 𝑑 𝑛×𝑘 𝑘×𝑘 𝑘×𝑑
Decoder
FeatureVectors
Output
Input
CNN Extractor
FC Layers
Logits
Input
Randomized SVD to obtain Top Left Singular Vector as the 𝓟 -vector
S a m p l e s
F ea t u r e s
Feature Vectors in CNN Classifiers and AE
Feature Extraction for Every Sample
T o p l e f t s i
ngu l
a r V e c t o r 𝓟 -vector
Encoder
GMM, (Richardson & Weiss, 2018) uncovered GAN’s superiority in feature/texture reconstruction and in the meanwhile its incompetence in distribution learning, while (Lee et al., 2019) provides some analytical insights on the structure of latent spaces.
In our work, we propose P -vectors to model the distribution of samples in the feature/latent spaces of DNN classiﬁers and AEs.
To the best of our knowledge, we make unique contributions compared to the above work.
M ETHODOLOGIES : F EATURE V ECTORS , S INGULAR V ALUE D ECOMPOSITION , AND P - VECTORS
As mentioned in Section.
1, we carry out extensive experiments on analyzing the feature vectors of DNN models with various architectures amd different training paradigms through the newlyproposed P -vectors.
In this section, we present the design of experiments in details, where we cover three key procedures of the experiment as shown in Figure 2.
Feature Vector Extraction.
Given a model, either the well-trained one or a checkpoint obtained during the training process, we extract the feature vector for every sample, with respect to the architectures.
For DNN classiﬁers (either under supervised/self-supervised learning), we use the output of CNN feature extractor (i.e., the input to the Fully-Connected Layer) as the feature vector of the given sample, while we vectorize the output bottleneck layer as the feature vector for AEs.
Note that, in our research, we consider AEs with symmetric architectures of encoders and decoders only.
Singular Value Decomposition with Feature Vectors.
Given the feature vector for every sample, we form a #samples × #features matrix and perform SVD to obtain the top left singular vector as the P -vector.
Furthermore, there is no need to solve the singular vectors of the complete spectrum, as only the top singular vector is requested for P -vector estimation.
In this way, we propose to use Randomized SVD (Halko et al., 2011) that compresses the feature domain and approximate the low-rank structure of SVD for acceleration purpose.
Actually, we compare the numerical solution of Randomized SVD and Common SVD for ResNet-50 on CIFAR-10 dataset (#features=256 and #samples= 50,000), where we need to perform SVD on a #samples ×
#features matrix and the Pvector/top left singular vector should be with dimensions.
Compared to the vanilla SVD, around 109x (from 44.88 seconds to 0.41 seconds) speedup has been achieved by Randomized SVD while no signiﬁcant numerical errors having been found in the results.
P -vectors, Principal Subspace, and the Distribution of Samples in Feature Spaces.
The top left singular vector of #samples ×
#feature matrix represents the principal subspace (Jolliffe, 2002) that samples are distributed into a one-dimensional space (with the maximal variances) by the projection of principal component (Abdi & Williams, 2010) of features.
In this way, we can compare and measure the divergence between the feature spaces of two DNNs through assessing the angle between their P -vectors based on the same set of samples.
Discussion.
Given the same set of samples, P -vectors of the two DNNs should be in the equal length, as they are both the top singular vectors in the sample side.
Furthermore, we can easily measure their divergence via Cosine of two vectors.
A larger cosine of two vectors (e.g., close to 1.0) usually refers to the evidence that the two networks share subspace in their features.
Note that,
[Table]
R e s N e t 18 W i
d e - R e s
C o n v A E
D e n
o
i s e A E S
i m C L R
S u p C
o n 1 0.99 0.98 0.99 0.99 0.98 0.95 0.93 0.99 1 0.99 0.98 0.98 0.98 0.95 0.
[Figure]
in high-dimensional spaces, the chance of orthogonality between two random vectors appears more frequently, due to the curse of dimensionality (Pestov, 1999).
Thus, given a sample set such as
CIFAR-10 with more than 60,000 samples, when the cosine measure close to 1.0 or the angle between the two P -vectors (with 60,000 dimensions) is small, we can conclude that the two networks would share a subspace in the feature spaces in high conﬁdence.
U NCOVERING C OMMON F EATURE S UBSPACE WITH P - VECTORS
[Figure]
In this section
[Figure]
we further elaborate the phenomenons that we introduced in Figure 1 of Section 1, where we would incorporate more details using rich datasets.
Common Subspaces.
To demonstrate the existence of common subspaces, we propose to measure the angles between the P -vectors extracted from well-trained models using the same datasets.
For any two models, a small angle between the two P -vectors refers to the facts that the two models would project these samples to similar principal subspaces in feature spaces, as the
P -vectors are estimated as the top left singular vector of the feature matrix (i.e., #samples × #features)
SVD based on the same group of samples.
Figure.
1 (a)–(b) presents the cosine of angles between the P -vectors of well-trained models (trained with various architectures and tasks using CIFAR-10 dataset).
To generalize our observations, we propose to compare the P -vectors of models using the feature matrices based on the training and testing samples in Figures.
1 (a) and (b) respectively.
We carry out the same experiments on CIFAR-10 and
CIFAR-100 datasets, and present the cosine of angles between the P -vectors of well-trained models in Figure 3.
All experiments demonstrate that, with a relatively large cosine (close to 1 ), the models well-trained using the same dataset, no matter what types of architectures or whether labels has been used in the feature learning of various tasks, share a common subspace.
Convergence to the Common Subspace To understand the dynamics of feature learning that shapes the common subspace from scratch, we measure the change of angles over epochs between the P -vectors of training models (i.e., checkpoint in the epoch) and well-trained ones in comparisons.
In Figure 1 (c)–(h), for every model of various tasks, we present angles between the P -vectors of the well-trained model and itself’s training checkpoint per epoch.
A consistent convergence could be observed.
As the P -vectors of well-trained models are close to each others (see in Figuresand 3), we can conclude the feature vectors extracted by these models would evolve on time and gradually converge to share the common subspace during the learning procedure.
[Figure]
We carry out a more comprehensive model-to-model comparison using
CIFAR-10 dataset.
Figures.
4 (a)–(c) present the convergence of P -vector angles between the well-trained supervised models and the checkpoints per epoch of supervised, unsupervised, and self-supervised learning models , where we use the welltrained Wide-ResNet28 (trained with 200 epochs under suggest settings) as the reference of supervised models.
Figures.
4 (d)–(f) present
P -vector angles between the well-trained ConvAE (trained with 100 epochs under suggest settings as the reference of unsupervised learning) and the checkpoints per epoch of supervised, unsupervised, and self-supervised learning models.
Figures.
4 (g)–(i) present the P -vector angles between the well-trained SimCLR representations (trained under suggest settings (Chen et al., 2020) as the reference of self-supervised learning) and the checkpoints per epoch of supervised, unsupervised, and self-supervised learning models.
The trends of convergence in all comparison further validate our hypotheses.
Discussion.
Note that, to better visualize every comparison, we use the maximal angles achieved during the ﬁrst epoch to represent the angles between P -vectors corresponding to the ﬁrst epoch in
Figures 1 and 4.
Actually, in the ﬁrst epoch, there would incorporate some non-monotonic trends for the angles varying over the number of iterations.
Figure 5 present the angles between the Pvectors of the training and well-trained models over the number of iterations in the ﬁrst epoch in three settings of learning, where we use CIFAR-10 dataset for the experiments.
In this way, we recover the procedure that the learning procedure shapes the feature space – with the random weight initialization, the angles of P -vectors between the initial models and the well-trained models are large, the angles drop down and rebound in a zigzag trend quickly in all three learning paradigms.
In the rest of learning procedure, the angles between the P -vectors drop down and converge to smaller ones over the number of iterations.
( H.II )
C ASE S TUDIES ON P - VECTORS WITH D ATA D ISTRIBUTIONS AND M ODEL P ERFORMANCE
[Figure]
In this section, we intend to extend our observations to connect Pvector with the (raw) data distributions and the generalization performance of models.
Correlation to Data P -vectors.
Given the raw data matrix, i.e., a #samples × #data dimensions matrix, we obtain the Data P -vector 2 of these samples using the top left singular vector of the raw data matrix, which represents the principal subspace of data (or the position of every sample projected by the principal component of the data).
To understand the connection between models and data, We carry out the case study using CIFAR-10 dataset, so as to measure the angles between the model P -vector and the data P -vector and how the angles change over the number of training epochs.
Figure.6 shows the convergence of P -vector angles between the raw data and the checkpoints per epoch of supervised, unsupervised, and self-supervised learning models.
The angles between the
[Figure]
data and model P -vectors starts from about orthogonal and generally decrease and converge to about 50 ◦ degree to 60 ◦ degree.
Note that the angle points corresponding to the ﬁrst epoch on the curves are the largest P -vector angles during the training procedure of the ﬁrst epoch from the random scratch.
The results indicate that the principal subspace of the well-trained models are more close to the principal subspace of the raw data, no matter what type of architectures are used for what kind of learning tasks and whether the labels are used for training.
Correlation to Deep Learning Performance.
We further explore the relationship between the performance of the various models on different datasets and the angles between P -vectors and data P -vectors.
Experiments are delivered using ResNet-20/56/110 and VGG-16 on CIFAR-10/ datasets.
As shown in Fig.7, there exists a strong and consistent correlations between the training/testing accuracy of models and the angles between the model and data P -vectors.
Note that in these experiments, we use the samples in the training dataset to estimate the model and data
P -vectors while avoiding the use of validation information, so as the demonstrate the connection of
P -vectors all based on training samples to the generalization performance.
Note that for CIFARdataset, we use ResNets with pre-activation enabled.
We also conduct correlation analysis via Spearman’s coefﬁcients and p-values.
With 0 .
as the threshold for signiﬁcance, we ﬁnd signiﬁcance in the correlations between the angles between model and data P -vectors and the training/testing accuracy for all above cases, expect the correlation between the angles and training accuracy on ResNet-100 using CIFAR-100 dataset (p-value= 0 . 0927 ).
Further for all models on all datasets, the correlation between angles and the testing accuracy is slightly higher than the correlations between angles and training accuracy (and signiﬁcantly lower p-value).
In this way, we could conclude that (1) both training and testing accuracy are correlated to the angles between the model and data P -vectors, (2) the strong correlations between angles and the testing accuracy might not be caused by the correlations between the angles and training accuracy, as the earlier ones are even stronger, (3) the angles between the model and data P -vectors would be a reasonable performance indicator, as they are strongly, consistently, and signiﬁcantly correlated to the testing accuracy.
Applications to Generalization Performance Prediction.
To further demonstrate the feasibility of using the P -vector as a “validation-free” measure of generalization performance, we use the experiment settings of “Predicting Generalization in Deep Learning Competition” at NeurIPS 2020 to
[Table]
evaluate P -vectors on predicting the generalization performance of a wide range of models using the training dataset.
The competition offers a large number of deep models trained with various hyperparameters and DNN architectures, while the ofﬁcial evaluator for the competition ﬁrst predicts the generalization performance of every model using the proposed measure, then verify the prediction results through the mutual information (the higher the better) between the proposed measures and the (observable) ground truth of generalization gaps.
In the experiments, we propose to use the angles between the model and data P -vectors using the training dataset as the metrics of generalization performance.
In the comparisons with the proposed
P -vectors, we include several baseline measures in generalization performance predictors, including
VC Dimension (Vapnik, 2013), Jacobian norm w.r.t intermediate layers (Jiang et al.)
, Distance from the convergence point to initialization (Nagarajan & Kolter, 2019), and the Sharpness of convergence point (Jiang et al., 2019).
In addition to these methods, we also propose “Pseudo Validation
Accuracy” as a measure for comparisons, where this measure ﬁrst uses random data augmentation apply to the original set of training data to generate a set of “pseudo validation samples”, then tests the accuracy of the model using “pseudo validation samples”.
Table 1 presents the comparisons between the proposed measures and baselines.
It shows that when the proposed measure – angles between the model and data P -vectors – stands alone, the measures signiﬁcantly outperforms the baseline methods including Jacobian norm w.r.t intermediate layers and the VC dimensions.
However, through complementing with other metrics, the metrics based on
P -vector angles could be further improved in predicting the generalization performance and ﬁnally outperform all baseline methods when combining with “Pseudo validation accuracy”.
Note that we combine the results of two metrics through weighted aggregation (Pihur et al.
(2009), with a constant weight 0.05) of two ranking lists that are sorted according to the two metrics respectively.
All above experiment results in three case studies backup our hypothesis that the P -vectors of the model feature spaces would connect to the data distribution and performance of models ( H.III ).
C ONCLUSION
In this work, we propose the P -vector, deﬁned as the top singular vector of the feature matrix (the #samples × #features matrix) through SVD, to characterize the principal subspace of the feature space founded by the samples.
We observe that, no matter what type of DNN architectures or whether the labels have been used to train the models, the P -vectors of different models would converge to the same one (or the cosine measure of the P -vector angle between any two models is close to 1), when the models are well trained with the same dataset.
We conclude that the feature spaces all well-trained DNN models using the same training dataset would share principal subspace.
Furthermore, during the training procedure from the random scratch, the model P -vector would slowly approach to the data P -vector (deﬁned as the top singular vector of the raw data matrix, i.e., #samples × #data dimensions), where the data and model P -vectors start from an almost-orthogonal status while the angles between the data and model P -vectors overall decrease and converge to smaller ones (50 ◦ –60 ◦ ).
Finally, we ﬁnd that angles between the model and data P -vectors are strongly correlated to the performance of models (i.e., both training/testing accuracy) while they are capable of predicting generalization performance, even when the model and data P -vectors are all estimated using training dataset only.
A. C OMPARISON OF ANGLES BETWEEN MODEL AND WELL - TRAINED P - VECTOR U SING D IFFERENT A RCHITECTURES ON C IFAR -
In the main text
[Figure]
we presented the result on CIFAR-10 dataset.
To generalize the observations
[Figure]
we repeated the experiments on CIFAR-100 dataset to validate our hypothesis of the convergence of the angles between model checkpoints and well-trained model P -vectors.
We investigate the change of angles over the P -vectors of training model checkpoints per epoch with comparison to the Pvectors of well-trained models (model of epoch 200 in our case).
As shown in Fig.8, a gradually decreasing manner of the curves for the angle between P -vectors and all angles between P -vectors cross models with different supervisory manners generally converge to a value that smaller than 10 ◦ degree.
We can conclude that the hypothesis of the existence of common subspace during the learning procedure also stands on the experiments with CIFAR-100 dataset
[Figure]
A. T HE M ON - MONOTONIC T REND WITHIN THE F IRST E POCH
The variation of angles between P -vectors for the well-trained model and its checkpoint per training epoch of each iteration in the ﬁrst epoch.
The non-monotonic trends within the ﬁrst epoch also incorporate in the experiments on CIFAR-100 datasets.
Fig.9 shows the curves indicating the variation of angles between the training model P -vectors and the well-trained model P -vectors in the iterations in the ﬁrst epoch.
As we use 128 as the batch size in training procedure, the number of iterations for updates is 391 per epoch.
We obtain the observation of a non-monotonic trend that the angle ﬁrst rises with the random initialization and drop down.
And in the rest of training process, the angles keeps the approximately monotonically decreasing and converging to small values.
The experiments shows consistent result and conclusion on CIFAR-100 dataset with the discussion in section 4.
A. M ODEL - TO - MODEL C OMMON S UBSPACE
We also test and verify the model-to-model common subspace shared by models trained with different supervisory manners on CIFAR-100 dataset.
Experiments carried out to evaluate the angles between P -vectors for checkpoints of all models and P -vectors for well-trained supervised, unsupervised and self-supervised models, where we use the well-trained Wide-ResNet28/Convolution
[Figure]
[Figure]
Auto-encoder and SimCLR model (trained with 200 epochs under suggest settings) as the reference of supervised, unsupervised and self-supervised models, respectively.
As shown in Fig.10, a consistent convergence for the curves of the angles can be observed and support our hypothesis that the dynamics learning procedure construct the common subspace gradually
[Figure]
A. T HE N ON - MONOTONIC T REND IN THE F IRST E POCH OF C OMPARISON OF A NGLES BETWEEN M ODEL AND R AW D ATA P - VECTORS
We also explore the construction procedure for the common subspace share between feature vectors and the raw data during the training process.
Experiments are carried out to compare the space of models and raw data P -vectors on the training dataset.
As shown in Fig.11, we observe a nonmonotonic trend that the angle ﬁrst rises with the random initialization and drop down.
The angles keeps the approximately monotonically decreasing and converging to small values in following training epochs.
The experiments shows consistent result on both CIFAR-10 and 100 dataset.
Note that we follow the default random data augmentation policy to pre-process the training dataset
[Figure]
A. C ASE S TUDIES ON THE A NGLE V ARIETY B ETWEEN M ODEL AND R AW D ATA FOR E ACH L AYER
To explore the dynamic variation process by zooming in to every layer variation in each epoch, we perform the case study using Resnet-18 structure on CIFAR-10 dataset.
As shown in Fig.12, we give the angles according to layers of 5 different epochs, where the x-axis indicating the indices of residual blocks in the network structure and y-axis refers to the angles between the model checkpoint and raw data P -vectors.
We observed that in early training stage, the angles between the model checkpoint and raw data P -vectors keeps an increasing manner when the features passing though layers and turn into a decrease trend towards the stacked layers in the late training stage.
This set of experiments further support our hypothesis that the dynamics learning procedure construct the common subspace gradually through training process.
A. A NALYSIS FOR D ISTRIBUTION OF THE P - VECTORS OF F EATURE M ATRICES
The following experiments are carried out on Cifar-10 dataset with Resnet 50.
[Figure]
[Figure] [Figure] [Figure]
[Figure] [Figure] [Figure]