ATD : Augmenting CP Tensor Decomposition by Self Supervision
Anonymous Author s)
Abstract
Tensor decompositions are powerful tools for dimensionality reduction and featureinterpretation of multidimensional data such as signals.
Existing tensor decomposition objectives (e.g., Frobenius norm) are designed for fitting raw data understatistical assumptions, which may not align with downstream classification tasks.
In practice, raw input tensor can contain irrelevant information while data augmentation techniques may be used to smooth out class-irrelevant noise in samples.
Thispaper addresses the above challenges by proposing augmented tensor decomposition ( ATD ), which effectively incorporates data augmentations and self-supervisedlearning (SSL) to boost downstream classification.
To address the non-convexityof the new augmented objective, we develop an iterative method that enables theoptimization to follow an alternating least squares (ALS) fashion.
We evaluate ourproposed ATD on multiple datasets.
It can achieve 0 .
8% ∼ 2 . 5% accuracy gain overtensor-based baselines.
Also, our ATD model shows comparable or better performance (e.g., up to 15% in accuracy) over self-supervised and autoencoder baselineswhile using less than 5% of learnable parameters of these baseline models.
Introduction
Extracting unsupervised features from high-dimensional data is essential in various scenarios, suchas physiological signals (Cong et al., 2015), hyperspectral images (Wang et al., 2017) and fMRI(Hamdi et al., 2018).
Tensor decomposition models are often used for high-order feature extraction(Sidiropoulos et al., 2017), among these, CANDECOMP/PARAFAC (CP) decomposition is one ofthe most popular models.
The low-rank CP tensor decompositions (Kolda and Bader, 2009) assumethat the input data is composited by a small set of components, while the reduced features are thecoefficients that quantify the importance of each basis, which provides a compact representation.
Under this low-rank assumption, existing tensor decomposition objectives aim to fit individual datasamples with statistical error measures (Hong et al., 2020; Singh et al., 2021), e.g., Frobenius normor KL-divergence.
Though fitness is an essential principle for feature reduction, common objectivefunctions do not account for downstream tasks, e.g., classification.
Contrastive self-supervised learning (SSL) (He et al., 2020) is recently popular for unsupervisedfeature learning, which utilizes the class-preserving data augmentations (Dao et al., 2019)
andlearns an encoder that can filter out class-irrelevant information.
The optimization goal is to enforcealignments (Chen et al., 2020; Wang and Isola, 2020), ensuring that the anchor sample is closer to thepositive sample (which has the same latent class as the anchor sample) than to the negative sample(which is in a different latent class) in the embedding space.
In an unsupervised setting, positivesamples are given by data augmentations, while negative samples are hard to acquire (Arora et al.
He et al., 2020; Chen et al., 2020).
Also, previous models are mostly built on deep neuralnetworks, which are often black-box models with tens of thousands of learnable parameters.
This paper aims to incorporate both the fitness and alignment principles into CP tensor decomposition 1by augmenting the common fitness objective with a new self-supervised loss.
The new self-supervisedloss is based on the unbiased estimation of negative samples Chuang et al.
(2020), which effectivelyprevents the sampling bias issue (Arora et al., 2019).
The purpose of our design (i.e., introducing
SSL into CP tensor decomposition) is to learn class-aware tensor decomposition results for boostingthe downstream tensor sample classifications.
To address the non-convex subproblems from the newobjective, we formulate an iterative method, which solves least squares optimization in each step withclosed-form solution.
Main contributions are summarized below.
• We propose augmented tensor decomposition , named ATD , which learns an unsupervised CPstructure decomposition by extending the original fitness objective with a self-supervised loss onthe contrastiveness of similar and dissimilar tensor samples.
• We develop an iterative method to address the non-convex subproblem from the new objective,which enables our algorithm to follow an alternative least squares fashion .
Our algorithm hasasymptotically the same complexity of each optimization sweep as the common CP-ALS.• We provide extensive evaluations on four real-world datasets and compare to recent tensordecomposition models, autoencoder models, and self-supervised models.
Our method shows better or comparable prediction performance in various downstream classifications while only requiring
much fewer (e
g., less than 5% of) parameters than that of deep learning baselines.
Background
Notation.
We use plain letters for scalars, such as x or X , boldface uppercase letters for matrices,e.g., X , boldface lowercase letters for vectors, e.g., x , and Euler script letters for tensors, randomvariables of tensors, and probability distributions, e.g., X .
Tensors are multidimensional arraysindexed by three or more indices (modes).
For example, an N -mode tensor X is an N -dimensionalarray of size I 1 × · · · × I N , where x i 1 ,...,i N is the element at the ( i 1 , · · · , i N ) -th position.
For matrix
X , the r -th row and column are x ( r ) and x r respectively, while x ij is for the ( i, j ) -th element.
∥ X ∥ Fis the Frobenius norm.
For vector x , the r -th element is x r , and we use ∥ x ∥ 2 to denote the vector2-norm, ⟨· , ·⟩ for the vector inner product, ◦ for the outer product, and (cid:74) · (cid:75) for the Kruskal product.
Indices in the paper start from 1 , e.g., x 1 is the first column of the matrix.
2.1 Tensor Modeling
This paper aims to learn tensor bases from unlabeled data and then use the bases to build a featureextractor for downstream classification.
Without loss of generality (w.r.t.
tensor order), we considerthe fourth-order tensor, e.g., a collection of multi-channel Electroencephalography (EEG) signals,
[Equation]
where T ( n ) ∈ R I × J × K .
The first dimension of T corresponds to data samples (e.g., one for eachpatient), while the other three are feature dimensions (e.g., channel by frequency by timestamp ).
Data Model.
To model the above tensor, previous works (Kolda and Bader, 2009) assume that
• There are a set of rank-one tensor components
E
= {E 1 , . . .
, E R } , which are learnable;•
The tensor data sample/slice T ( n ) admits a low-rank structure and can be represented as a weightedsum of these tensor components E , where x ( n ) denotes its coefficient vector;• On top of the low-rank structure, each data sample T ( n ) also contains additional element-wise i.i.d.
Gaussian noise due to real-world distortion (e.g., physical noise in signal measurements).
In the context of downstream classifications, we further assume that each sample T ( n ) is semanticallyassociated to one of the latent classes p ∈ { 1 , . . .
, P } , and we let D p be the sample distribution ofclass- p .
Thus, the data sample can be formulated as, ∀ n ,
[Equation]
where
[Equation]
σ
[Equation]
, where σ is generally small .
Here a r , b r , c r are the learnable parameters, which produces the rank-one component E r , and theyare the column vectors of { A ∈ R I × R , B ∈ R J × R , C ∈ R K × R } , referred as "bases".
CANDECOMP/PARAFAC Decomposition (CPD).
Given the above tensor model, standard CPDonly captures the i.i.d.
Gaussian noise by minimizing the negative log-likelihood (NLL), whichresults in the following standard fitness/reconstruction loss,
[Equation]
Here, the Kruskal product (cid:74) · (cid:75) outputs a fourth-order reconstructed tensor from four input factor 86 matrices.
For consistency, if the first input is a vector, the output is considered as a third-order tensor.
2.2 Problem Formulation
CP decomposition seeks a low-rank reconstruction, without special consideration for the downstreamtask.
In this paper, we are motivated to improve the CPD model by exploiting the latent classes (in anunsupervised way) and learn good bases to provide better class-aware features for classification.
What are Good Bases?
This paper considers two design principles for good bases.
The firstprinciple is fitness , which requires a low-rank tensor reconstruction with the bases.
Second, datasamples associated with the same latent class should be decomposed into similar coefficient vectors,with the bases, while the vectors should be dissimilar if the samples are from different latent classes.
This principle is called alignment , which is important for classification but not considered in thestandard tensor decomposition.
In this paper, we assess the quality of the learned bases by theperformance of downstream classification, where the coefficient vectors (obtained using the bases viadecomposition) are the feature inputs (into the downstream classifier).
Working Pipelines.
To put it succinctly, the paper tackles an unsupervised learning problem whileusing downstream supervised classification for evaluation.
The procedures are briefly outlined:
• First, we learn the bases { A , B , C } from a large set of unlabeled data.
The loss function is developed in consideration of the fitness and alignment (defined in the next section)
principles.•
Then, we construct the following feature extractor given { A , B , C } .
The feature vector of anew sample is obtained by the closed-form solution of the least squares problem ( α > 0 is a hyperparameter),
[Equation]
Note that, when f ( · ) is applied to a batch of samples, it outputs a coefficient matrix.
• Next, we evaluate the feature extractor with a set of labeled data.
Given f ( · ) , we first apply it on the labeled data to extract their features and then train an additional logistic regression model (asthe downstream classifier) on top of the extracted features, so that the result of classifications will
implicitly reflect how good the bases are.
Augmented Tensor Decomposition
We show our model in Figure 1.
The design is inspired by the recent popularity of SSL.
To exploitthe latent class information, we introduce class-preserving data augmentation into CPD model anddesign self-supervised loss to constrain the learned low-rank features (i.e., the coefficient vectors).
Data Augmentation 2 .
In general, data augmentation methods are chosen to perturb the raw data whilepreserving class label information.
Given a sample T ( n ) , we assume that after data augmentation, its perturbation ˜ the the component 3.1 Self-supervised Loss
The design of our self supervised loss corresponds to the alignment principle, which is based onpairwise feature similarity and dissimilarity.
We call a pair of data samples from the same latentclass as positive pair , a pair of samples from different latent classes as negative pair and a pair ofindependent samples (two random samples from the dataset) as random pair .
Intuitively, an anchorplus a positive sample composes a positive pair, similarly for negative pairs and random pairs.
In thiswork, our self-supervised loss aims to maximize the feature similarity between positive pairs andminimizes that between negative pairs in an unsupervised way (no labels during optimization).
Formally, let X p , Y p be discrete random variables (of tensor samples) distributed as D p , p ∈{ 1 , . .
. , P } .
We want to minimize the following objective when no class labels are given ,
[Equation]
where λ ≥ 1 is a hyperparameter and
[Equation]
Here L pos maximizes the feature similarity of positive pairs while L neg minimizes the featuresimilarity of negative pairs.
f ( · ) is the feature extractor, defined in Eqn.
(2), and the similaritymeasure is given by cosine distance, parameterized by two random variables,
[Equation]
To this end, the key is to implement the self-supervised loss, i.e., Eqn.
(3), in an unsupervised setting.
Specifically, we want to construct the sampler of positive pairs and the sampler of negative pairswith unlabeled data only.
The sampler of positive pairs (in short, positive sampler) can be easilyapproximated by data augmentation techniques, which provides "surrogate" positive pairs (given anysample as the anchor, we apply data augmentation methods to generate a perturbed data as positivesample, and then the anchor plus the perturbed data is a positive pair).
However, the negative sampleris infeasible, without labels.
As a compromise, previous works He et al.
; Chen et al.
(2020)consider using random sampler to replace the negative sampler given that the random sampler canbe easily achieved by picking two independent samples from the dataset.
However, this practice isshown to induce sampling bias and hurts the learned representation Arora et al.
(2019); Chuang et al.(2020) since a random pair may be from the same latent class.
Construction of Negative Sampler.
In this paper, we consider using the law of total probabilityto construct the negative sampler in a statistical way.
Formally, assume r p is the label rate of latent
class- p (thus, we have (cid:80) p r p = 1 ), we apply the law of total probability and the following holds,
[Equation]
Here, the usage of E [ · ] means that the expectation is taken over four interdependent random variables,i.e., p, q, X p , Y q , while E p,q [ · ] means that p, q is fixed and thus it is only taken over two randomvariables, i.e., X p , Y q .
The result shows that the negative sampler can be equivalently replaced by aweighted combination of the random sampler and positive sampler.
Here we do not have access tor p , ∀ p with unlabeled data, this issue is dealt with later.
Self-supervised Loss.
Consequently, we can reformulate our self-supervised loss as, L ss = L pos + λ L neg
[Equation] [Equation]
From Eqn.
(5) to Eqn.
(6), we use the results in Eqn.
Two-sided Bound.
The above form still requires label rate information, i.e., r p , ∀ p , we thereforeconsider using the following approximation to the above loss L ss ,
[Equation]
Here, γ ≥ 0 is a hyperparameter, while L ss can be bounded as (derivations in appendix E),
[Equation]
The equivalence is established when C 1 = C 2 , i.e., the class labels are balanced.
To simplify thederivation, we ignore λ in the following and let γ be a new hyperparameter.
Also, the constants
C 1 and C 2 can be absorbed into a weight hyperparameter β , given in the next section.
This boundimplies that, an easy-to-compute β L Θ ss ( γ ) is often a good approximation of L ss for some β .
The nextsection specifies how to compute β L Θ ss ( γ ) unsupervisedly as our empirical self-supervised loss .
3.2 The Objective of ATD
Empirical Estimator.
We obtain an empirical estimator of L Θ ss with Monte Carlo method.
Suppose
T and ˜ T are the input tensor and the augmented tensor respectively, and X = f ( T ) , ˜ X
= f ( ˜ T ) [-
-]∈
R N × R are the coefficient/feature matrices.
We use the row vectors of X , ˜ X to estimate Eqn.
The first term E [ sim ( f ( X p ) , f ( Y q ))] essentially requires a random sampler, which is approximatedby the average cosine similarity of all possible pairs of non-corresponding row vectors of X , ˜ X , whilethe second term E [ sim ( f ( X p ) , f ( Y q ))
| p = q ] requires a positive sampler, which is estimated bythe average cosine similarity of all pairs of corresponding row vectors,
[Equation]
where D ( X ) = diag (cid:16)
[Equation]
[Equation]
is the row-wise scaling matrix and
[Equation]
 .
Note that, the form in Eqn.
(9) is significantly different from tensor discriminant analysis (Jia et al.
Tao et al., 2007), which integrates the actual label information as a regularizer and is alsodifferent from graph regularized tensor decomposition (Cai et al., 2010), which incorporates sideinformation, such as geometrical positions (Maki et al., 2018).
Compared to standard noise contrastiveestimation (NCE) (Gutmann and Hyvärinen, 2010; Chen et al., 2020) in the area of contrastive SSL,our SSL form in Eqn.
(9) considers a subtraction form instead of the softmax formulation, making itamenable to quadratic optimization, as we will show in Sec. 3.3.
Overall Objective.
According to Eqn.
(8), the self supervised loss L ss is bounded by L Θ ss ( γ ) , whilethe constants (i.e., C 1 , C 2 ) can be absorbed into a weight hyperparameter β .
We let the empirical selfsupervised loss, ˜ L ss =
β ˜ L Θ ss ( γ ) .
Our objective follows both the fitness (i.e., CPD reconstruction loss)and alignment (i.e., self-supervised loss) principles, while also considering Tikhonov regularization(Golub and Von Matt, 1997) to constrain the scale of all parameters,
[Equation]
where
[Equation] [Equation]
The objective has (i) three hyperparameters, γ, α, β > 0 ; (ii) five factor matrices, A , B , C , X , ˜ X .
3.3 Alternating Least Squares Optimization
Ideally, we would like to use alternative least squares (ALS) algorithm (Sidiropoulos et al., 2017)for optimizing the objective, where each factor matrix is updated in a sequence by solving leastsquares subproblems.
With large scale tensors, we can also resort to mini-batch stochastic ALS (Caoet al., 2020) to reduce memory footprint of the decomposition.
However, the objective in Eqn.
(11)is non-convex to X and ˜ X , respectively.
For addressing the non-convex subproblem, we design aniterative method in this section, which only involves solving least square problems.
Addressing Non-convex Subproblem.
We use the subproblem of X as an example.
Given A , B ,
C , ˜ X fixed, we want to minimize Eqn.
(10) by finding the optimal solution, denotd as X ∗ , [Equation] X
[Equation]
• First, we are interested to find that the matrix-formed problem in Eqn.
(12) can be decomposed intorow-wise subproblems and to obtain the solution of Eqn.
(12), it is suffice to solve each subproblemindependently.
Let us consider the subproblem of the k -th row, which is
[Equation]
where T ( k ) is the k -th slice of T , and g ( k ) is the k -th row of G ( γ ) .•
Here, Eqn.
is still non- convex.
We let the derivative of Eqn
) objective to be zero andarrange the terms, which yields,
[Equation]
where
[Equation]
Here x , v 1 , v 2 are row vectors and V 3 is a matrix.
T ( k ) 1 is the 1 -mode unfolding of T ( k ) , ⊙ is the
Khatri-Rao product and ∗ is the Hadamard product (i.e., element-wise product).•
We consider the following iterative rule and the fixed point is a solution for Eqn.
(14), which is astationary point of Eqn.
[Equation]
We use an initial guess x init (obtained by solving Eqn.
(13) with while β = 0 , which is a leastsquare problem) to start
and then we repeat Eqn.
(15) for each row (i.e., each k ) and let the improvedguess be the initial guess, x init ← x impr , to iteratively improve the result.
Theorem 1 (proof in appendix A) ensures that Eqn.
(15) converges linearly if β is chosen to besufficiently small.
In appendix B, we verify the liner convergence and also empirically show that oneround of Eqn.
(15) is sufficient in our experiment, where β = 2 .
The non-convex subproblem of ˜ Xcan be solved in the same way.
Theorem 1.
Given non-zero row vectors v 1 , v 2 , u 0 ∈ R d , non-singular matrix V 3 ∈ R d × d andβ > 0 .
The sequence { u t } , generated by u
[Equation]
=
[Equation]
, satisfies,
[Equation]
where u ∗ is the fixed point and m = min t ∥
u t ∥ 2 , M = max t ∥ u t ∥ 2 are the bound of the sequence.
With a good u 0 and a sufficiently small β , we can safely assume [-
-]0 < m ≤ M < ∞ .
Optimization Procedures.
To minimize Eqn.
(11), we alternatively update A , B , C , X , and ˜ X ,where each subproblem involves only solving least squares problems with closed-form solutions.
With large-scale tensors (as in the experiments), we optimizes the factors in mini-batches.
Betweenmini-batches, the basis factors A , B , C are shared and updated incrementally.
We summarize thealgortihms in appendix C. The computation head of the algorithm is matricized tensor times KhatriRao product (MTTKRP).
The complexity of our optimization algorithm is asymptotically the sameas applying CP-ALS, which costs O ( NIJKR ) to sweep over the whole tensor once.
Experiments
This section presents the experimental evaluations.
Due to space limitation, additional details,including data augmentations and baseline implementation, are presented in appendix D.
4.1 Experimental Setup
Data Preparation.
We use four real-world datasets: (i) Sleep-EDF
(Kemp et al., 2000), whichcontains EOG, EMG and EEG Polysomnography recordings; (ii) human activity recognition (HAR)(Anguita et al., 2013) with smartphone accelerometer and gyroscope data; (iii) Physikalisch Technische Bundesanstalt large scale cardiology database (PTB-XL)
(Alday et al., 2020) with 12-lead ECGsignals
; (iv) Massachusetts General Hospital (MGH) (Biswal et al., 2018) datasets with multi-channel
EEG waves.
All datasets are split into three disjoint sets (i.e., unlabeled, training and test) by subjects,while training and test sets have labels.
Basic statistics are shown in Table 1.
All models (baselinesand our ATD ) use the same augmentation techniques: (a) jittering, (b) bandpass filtering, and (c) 3Dposition rotation.
We provide an ablation study on the augmentation methods in appendix D.5.
Baseline Methods.
We include the following comparison models from different perspectives:
• Tensor based models : ATD ss − is our variant, which removes the self-supervised loss from theobjective in Eqn.
(10); Stochastic alternating least squares (SALS) applies on the the CPD objectivewith Tikhonov regularizer, which works on large tensors; Graph regularized SALS (GR-SALS) augments the objective of SALS with a graph regularizer (Maki et al., 2018;
Cai et al. , 2010),define as Tr (cid:0) X ⊤ GX (cid:1) .
[Table] [Table]
• Self-supervised models : SimCLR - r
(Chen et al., 2020) and BYOL - r (Grill et al., 2020) are twopopular SSL models with their own objective functions, where r indicates the size of the outputrepresentation.• Auto-encoder models : AE - r denotes a CNN based autoencoder with mean square error (MSE)reconstruction loss, and AE ss - r denotes the same autoencoder model with standard NCE loss inthe bottleneck layer, where r is the representation size.
All models use the unlabeled set to train a feature encoder and use training and test sets to evaluate.
Note that, deep neural network models use the same CNN backbone.
In appendix D.8, we have alsocompared with two recent supervised tensor learning models, which shows the usefulness of our ATDand the large unlabeled set, especially in low-label rate scenarios.
Evaluation and Environments.
We evaluate model performance mainly based on classificationaccuracy , where we train an additional logistic classifier (He et al., 2020) on top of the feature encoder.
Also, for different models, we compare their number of learnable parameters .
The experiments areimplemented by Python 3.8.5, Torch 1.8.0+cu111 on a Linux workstation with 256 GB memory, 32core CPUs (3.70 GHz, 128 MB cache), two RTX 3090 GPUs (24 GB memory each).
All trainingis performed on the GPU.
For tensor based models, we use R = 32 and implement the pipeline in
CUDA manually, instead of using torch-autograd .
4.2 Experimental Results
This section shows the experimental results on downstream classification.
We use all the unlabeleddata to train the encoder or feature extractor, and use training data (since Sleep-EDF and MGHdatasets have enough training samples, we randomly selected a subset of them) for learning adownstream classifier and use all test data.
Each experiment is conducted with five different randomseeds and the mean and standard deviations are reported.
The metrics are the accuracy and thenumber of learnable parameters .
All models have 32-dim features in the end, except that for twoself-supervised baselines and autoencoder models, which have 128-dim options.
4.2.1 Better Classification Accuracy with Fewer Parameters
From Table 2, ATD shows comparable or better performance over the baselines.
We have also reportedthe running time per epoch/sweep in appendix D.7 for all models.
Compared to the variant ATD ss − ,our ATD can improve the accuracy by 1 . 0% ∼ 1 . 9% , which shows the benefit of the inclusion ofself-supervised loss.
SALS and ATD ss − have similar performance, while their objectives differ in that
[Figure] ATD ss − considers the Frobenius norm of the augmented data.
Thus, their accuracy gap is caused bythe use of data augmentation.
Also, the experiments show that the fitness and alignment principlesare both important.
We observe that with a self-supervised loss (i.e., alignment ), AE ss can givesignificant improvement over AE, while ATD shows ∼ 8% accuracy gain over the self-supervisedmodels on MGH dataset, since we can better preserve the data with a reconstruction loss (i.e., fitness ).
Moreover, the table shows that tensor based models require fewer parameters, i.e., less than 5%of parameters compared to deep learning models.
On HAR, the deep unsupervised models showpoor performance due to (i) they may not optimize a large number of parameters on middle-scaledataset; (ii) movement signals in HAR might have few degrees of freedom, which matches wellwith the low-rank assumption of tensor methods.
On large-scale Sleep-EDF, self-supervised modelsoutperforms ATD marginally since they have more parameters thus can capture more information.
4.2.2 Better Performance in Low-label Rate Scenarios
On the MGH dataset, we also show the effect of varying the amount of training data in Figure 2.
Weinclude an end-to-end convolutional neural network (CNN) model based on (Biswal et al., 2018),called Reference CNN , which is a supervised model and only uses the training and test sets.
To bemore readable, we separate the comparison figure into two sub-figures: the left compares our ATDmodel with self-supervised and auto-encoder baselines and the right one compares ATD with tensorbaselines and the reference model, and the scale of y-axis on two sub-figures are the same.
We find that all unsupervised models outperform the supervised reference CNN model in scenarioswith fewer training samples.
With more training data, the performance of all models get improved,especially the reference CNN model, which can optimize the encoder and predictive layers in anend-to-end way and finally outperforms our ATD when more training samples is available.
4.2.3 Stable Results with Hyperparameter Variation
For a comprehensive evaluation, we also conduct ablation studies on the effect of the data augmentation methods and on hyperparameters.
Due to space limitation, we move the experimental settings andresults to appendix D.6, while summarizing the general conclusions here: (i) with more diverse dataaugmentation methods, the final results are relatively better; (ii) with a larger rank R , the performancewill be better generally; (iii) our ATD is not sensitive hyperparameters α, β, γ .
Conclusion
This paper introduces the concept of self-supervised learning for tensors and proposes Augmented
Tensor Decomposition ( ATD ) and the ALS-based optimization.
We show that by explicitly contrastingpositive and negative samples, the decomposition results are more aligned with downstream classification.
On four real-world datasets, we show the advantages of our model over various unsupervisedmodels and with fewer training data, our model even outperforms the reference supervised models.
Compared to deep learning methods, tensor based models are not as flexible in processing multimodaland diverse inputs, such as natural images.
However, applying tensor decomposition on the outputsof earlier layers of pre-trained deep neural networks may be a feasible way to address the weaknesses.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.•
Did you include the license to the code and datasets?
[N/A]Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] See Conclusion
(c) Did you discuss any potential negative societal impacts of your work? [No](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results? [Yes] see Theorem 1(b) Did you include complete proofs of all theoretical results
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See
supplementary
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)? [Yes] See appendix D (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] See Table
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators? [Yes] (b) Did you mention the license of the assets?
[Yes](c) Did you include any new assets either in the supplemental material or as a URL? [N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[Yes] See appendix D.
(e) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content? [N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable? [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A] A Proof of theorem Proof.
We provide the proof below (first by submultiplicativity),
[Equation]
Then
[Equation]
we decompose the last term into two parts and have ∥ u ∗
[Equation]
2 − ∥
u t ∥ 2 = ∥
u t ∥ 2 − ∥ u ∗ ∥∥ u ∗ ∥ 2 ∥ u t ∥ 2 ≤ ∥ u t − u ∗ ∥∥ u ∗ ∥ 2 ∥ u t ∥ 2 ≤ ∥ u t − u ∗
∥m, u t ⊤ u t ∥ u t ∥ 32 − u ∗⊤ u ∗ ∥
u ∗ ∥ 32 = (cid:18) u t ⊤ u t ∥
u t ∥ 32 − u ∗⊤ u t ∥ u t ∥(cid:19) + (cid:18) u ∗⊤ u t ∥ u t ∥ 32 − u ∗⊤ u ∗ ∥
u t ∥(cid:19)
+ (cid:18) u ∗⊤ u ∗ ∥
u t ∥ 32 − u ∗⊤ u ∗ ∥
u ∗ ∥(cid:19)
=
( u t ⊤ − u ∗⊤ ) u t ∥
u t ∥ 3+ u ∗⊤ ( u t − u ∗ ) ∥
u t ∥ 3+ ∥ u ∗ ∥ 22 ( ∥ u ∗ ∥ 32 − ∥ u t ∥ 32 ) ∥
u t ∥ 3 2 ∥ u ∗ ∥≤ ∥
u t − u ∗ ∥m+ M ∥
u t − u ∗ ∥m+ ( M + 2 m ) ∥
u t − u ∗ ∥m.
The proof is complete by triangular inequality.
B Analysis of the Iterative Rule
In this section, we study how many rounds of the iterative rules are needed to achieve a goodclassification result.
This study is conducted on HAR and Sleep-EDF with five random seeds.
Linear Convergence Speed.
First, we study the convergence speed (when β = 2 ). We consider twoscenarios: (Scenario 1) when the bases { A , B , C } are initialized as random matrices; (Scenario 2)when the based { A , B , C } are already learned. We use the average relative difference (of the F-norm)as the convergence measure, i.e., 1 N (cid:80) Nk =1 ∥ x ( k ) t +1 − x ( k ) t ∥
∥ x ( k ) t ∥ = 1 N
[Figure] ∥ x [Figure]
, where x ( k ) meansthe k -th row of X and t means the number of rounds of the iterative rule.
We test on t = 1 , 2 , 3 , 4 , 8 .
The comparison is shown in Figure 3.
Both scenarios verify the linear convergence speed.
[Figure]
One Round is Enough.
We consider the performance of downstream classification with differentrounds of iterative rules.
The results are shown in Table 3 and Table 4.
[Table]
[Table]
1 Input initialized hyperparameters
2 Obtain the augmented
; 3 repeat Use A , B , C , ˜ X to update X by our iterative rules (one iteration) in Eqn. [Table] Use A , B , C , X to update ˜ X by our iterative rules (one iteration) in Eqn.
[Table]
Use B , C , X , ˜ X to update A by solving least square problem;
[Table]
Use A , C , X , ˜ X to update B by solving least square problem;
[Table]
Use A , B , X , ˜ X to update C by solving least square problem;
[Table] 2 repeat the
; / ∗ a a batch T l ∈ R b × I × J × K augmentation ˜
T l do use A l , B l , C l to initialize X based
T l ; use A l , B l , C l to initialize ˜ X based on ˜ T l ;
Use A l , B l , C l , ˜ X to update X by our iterative
We observe that with an increasing number of rounds of iterative rule, the classification results willnot improve further, however, the time consumption increases.
Thus, we use only one round of theiterative rule in our experiments.
C Algorithm
For small tensors, the algorithmic pipeline is presented in Algorithm 1.
For large tensors, we usemini-batch alternating least squares optimization in Algorithm 2.
D.1 Dataset Processing Sleep-EDF is public, which contains 153 full-night EEG (from Fpz-Cz and Pz-Oz electrode locations), EOG (horizontal), and submental chin EMG recordings, under Open Data Commons Attribution
License v1.0 and MGH Sleep is provided by (Biswal et al., 2018), where F3-M2, F4-M1, C3-M2,
C4-M1, O1-M2, O2-M1 channels are used, containing 6,478 recordings.
These two EEG datasets are processed in a similar way.
First, the raw data are (long) recordings of each subject.
On subject-level,these recordings are categorized into unlabeled and labeled sets by 90% : 10% .
Then the labeled setsare further separated into training and test by 5% : 5% .
Next, within each set (unlabeled, training,test), recordings are further segmented into disjoint 30-second-long periods, which are the datasamples in our study. Each data sample is represented as a matrix, channel by timestamp , and theyare associated with one of five sleep stages, Awake (W), Non-REM stage 1 (N1), Non-REM stage 2(N2), Non-REM stage 3 (N3), and REM stage (R). HAR is also public, collected as 3-axial linearacceleration and 3-axial angular velocity at a constant rate of 50Hz by the embedded accelerometerand gyroscope. It has been randomly partitioned into 70% and 30%.
We use 70% as the unlabeleddata (labels removed) and split the other part: 15% as training data and 15% as the test data.
Thelicense of this dataset is included in their citation.
PTB-XL is a public ECG dataset, which contains21,837 clinical 12-lead ECGs (male: 11,379 and female: 10,458) of 10-second length with a samplingfrequency of 500 Hz.
We randomly split the dataset into unlabeled (remove the labels) and labeledsets by 90% : 10% ; then the labeled sets are further separated into training and test by 5% : 5% .
Thisdataset is under Open BSD 3.0.
All datasets are de-identified (e.g., no names, no locations), and there is no offensive content.
All thelabels are also provided along with the datasets.
The label distributions are shown below. [Table] [Table] HAR
Walk [Table] D.2 STFT Transform We find that directly using the raw data (spatial information) does not provide good results, even forthe deep learning models.
Thus, we take Short-Time Fourier Transforms (STFT) as a preprocessingstep.
From a single channel, we can extract both the amplitude and phase information, which is thenstacked together as two different channels.
After STFT, each data sample becomes a three-ordertensor, channel by frequency by timestamp .
The FFT size is 256 and hop length is 32 for Sleep-EDF;the FFT size is 64 and hop length is 2 for HAR; the FFT size is 256 and hop length is 64 for PTB-XL;and the FFT size is 512 and hop length is 128 for MGH.
We use these third-order tensors as finalinput data samples for all models.
D.3 Data Augmentation
In our work, we build our feature extractor f ( · ) from tensor decomposition tool, which may notbe as expressive/flexible as deep neural networks in SSL, and thus we also ask the augmentationmethods to allow a similar component-based representation for the perturbed data.
Use EEG signalsas an example, we consider jittering and bandpass filtering as two augmentation methods in theexperiments, which perturb the signal frequency information and will not significantly change thelow-rank structure of the data.
As mentioned in the main text, we consider three different augmentation methods: (i) Jittering addsadditional perturbations to each sample.
We consider both high and low-frequency noise on eachchannel independently.
For high-frequency noise, we first generate a noisy sequence s , which has thesame length as the signal channel, and each element of s is i.i.d.
sampled from a uniform distribution
U [ − 1 , 1] .
We then control the amplitude of s by the noisy degree d ∈ R .
Finally, we add the scalednoisy sequence d · s to the channel.
In the {+experiment, d = 0 . for Sleep-EDF, d = 0 . 
 for HAR,d = 0 . 001 for PTB-XL and d = 0 .+}
01 for MGH.
For low-frequency noise, we generate a short noisysequence (the length is randomly sampled from a uniform distribution U [100 , length of channel ] ) inthe same way and then use scipy.interpolate.interp1d to interpolate the noisy sequence to be at thesame length as the channel.
The choice of high-frequency noise or low-frequency noise, or both arecoin-tossed with equal probability.
(ii) Bandpass filtering reduces signal noise.
{+We use the order-
+}
Butterworth filter by scipy.signal.butter to preserve only the within-band frequency information.
The high-pass and low-pass are (1 Hz, 30 Hz ) and (10 Hz, 49 Hz ) for Sleep-EDF, (1 Hz, 20 Hz )and (5 Hz, 24 .
5 Hz ) for HAR, (1 Hz, 30 Hz ) and (10 Hz, 50 Hz ) for PTB-XL, (1 Hz, 30 Hz ) and(10 Hz, 50 Hz ) for MGH.
Low-pass or high-pass or both are selected with equal probability.
Also,the bandpass filtering is applied to each channel independently.
The intuition is that the low-passsignals and high-pass signals might be both useful.
(iii) 3D position rotation is an augmentationtechnique used only for HAR datasets, which have x-y-z axis information from accelerometer andgyroscope sensors.
We apply a 3D x-y-z coordinate system rotation by a rotation matrix to mimicdifferent cellphone positions.
All augmentation methods are applied in sequence (i) (ii) (iii).
The
STFT is performed after the data augmentation.
[Figure]
Since all deep learning baselines are based on CNN, we use the same backbone model, shown in
Figure 4.
The model is adopted from (Cheng et al., 2020).
Based on the backbone model, weadd a fully connected layer for the reference CNN model, add non-linear layers for self-supervisedmodels (they also have their respective loss), and add corresponding deconvolutional layers forautoencoder models.
The reference CNN models is end-to-end, and they are trained on the trainingset; other baselines learn a low-dimensional feature extractor from an unlabeled set, and then alogistic classifier is trained on the training set, on top of the feature extractor.
Without loss ofgenerality, we use 128 as batch size.
For deep learning models, we use Adam optimizer with alearning rate 1 × 10 − 3 and weight decaying 5 × 10 − 4 .
We use 2 × 10 − 3 as the learning rate forour ATD .
Since the paper deals with unsupervised learning and uses standard logistic regression( sklearn.linear_model.LogisticRegression ) to evaluate, the hyperparameters of all models (exceptsupervised model) are chosen based on the classification performance on the training data.
For our
ATD model, by default, we use R = 32 , α = 1 × 10 − 3 , β = 2 , γ = b (which is the batch size), andall the reference configurations for each experiments are listed in code appendix.
The choice of Rdepends on the trade-off between model fitness and time complexity.
Specifically, a larger R meansbetter fitness and more preserved information in the extracted representations, while the number oflearnable parameters and time complexity also increases linearly with R .
In our paper, we run simple
CP decomposition on a small subset of the tensor and monitor the fitness curve.
We find that thefitness does not improve much around R = 32 for all datasets (which means the real tensor mighthave a smaller rank and the residual part might be just noise).
Thus, we choose R = 32 throughoutthe paper.
Effect of High-quality Data Augmentations. We study the effects of high-quality data augmentation methods. Let us assume (a): jittering , (b): bandpass filtering , (c): 3D position rotation .
Wetest on different combinations of the augmentation methods.
The experiments are conducted on twodatasets: (i) for the MGH dataset, we use 50,000 unlabeled data, 5,000 training samples, and all testsamples; (ii) for the HAR dataset, we use all data.
We use 128 as the batch size for MGH Sleep, 64for HAR, and R = 32 as the tensor rank.
Table 6 and Table 7 conclude that the augmentation methods influence the final classification results.
However, for different datasets, the effects are different.
We observe that for the MGH Sleep dataset,
[Table] [Table] [Table]
bandpass filtering works better than jittering.
In HAR, combinations of augmentations work betterthan the individual augmentations. Overall, we find that with more diverse data augmentation methods,the final results are relatively better. The study of how to choose/design (or even automaticallygenerate) better augmentation techniques will be our future work. Impact of Low-quality Data Augmentations.
We also study the impact of low-quality dataaugmentations.
We use the SALS model (mini-batch tensor baseline without data augmentation) andour ATD as the reference models and conduct the following experiments on MGH and HAR: • MGH-d: we change the d values for the jittering data augmentation, d means the ratio of the amplitude of the high/low frequency noise over the amplitude of the signal. • MGH-(A): on MGH data, we set the degree of the jittering method to an unrealistic value d = 1 as
the low-quality augmentation method, meaning that the magnitude of the noise is the same as the
magnitude of the signals.
We keep the bandpass filtering unchanged.
• MGH-(B): on MGH data, we randomly create a bandpass filter from (1 Hz, 10 Hz ) or
(30 Hz, 50 Hz ) as the low-quality augmentation method, which drops the critical middle-band
information.
We keep the jittering unchanged.
• MGH-(C): on MGH data, we combine the above two low-quality augmentation methods. • HAR-(A)(B)(C): follow the similar low-quality data augmentation method design on HAR. For (B), we use (1 Hz, 5 Hz ) and (20 Hz, 24 .
5 Hz ) as the low-quality bandpass.
The performances are shown in Table 8.
We find that low-quality data augmentations will hurt thelearned tensor bases and hinder the downstream classification.
With d changing from 0 .
02 to 5 , thejittering method becomes more unrealistic and the generated samples deviate from the real signaldata distribution.
Thus, we can find that the final classification performance also becomes worsegradually.
If all data augmentation methods are of low-quality, the performance cannot surpass thebase SALS model.
We also find that the performance drop is not significant.
The reason mightbe that even the augmentation methods are of low-quality, the Frobenius loss can still enforce themodel to learn decent subspaces for better fitness and find generic low-rank features (opposed to theclassification-oriented low-rank features), in this case, the features only follow fitness principle notthe alignment principle, so the performance will be similar compared to SALS.
[Figure]
[Figure] [Figure] [Figure] [Figure]
Choices
[Figure] gamma 0. C l a ss i f i c a t i o n A cc u r a c y ( % )
Varying gamma on HAR
[Table] [Table]
D.6 Ablation Studies on Hyperparameters
This section conducts ablation studies for decomposition rank R and other hyperparameters, α , β , γ .
The experiments are conducted on Sleep-EDF with 50,000 random unlabeled data, 5,000 randomtraining samples, and all test samples, and the HAR dataset.
The results are shown in Figure 5.
First, we can conclude that with a larger decomposition rank R , theperformance will be better generally. Though we observe that the performance worsens from R = 64to R = 128 , with limited training data, if the representation size (equals to R ) becomes larger, thelogistic regression model can overfit. We compare our model to other tensor-based methods with
R = 8 , 16 , 32 , 64 , 128 and the Table 9 and Table 10 shows that ATD outperforms the tensor baselinesconsistently with different R. Also, we find that the choices of α do not affect the final performance a lot. Finally, we find thatit is easy for users to select the β values from a large range in general. For example, selecting aβ ∈ [5 , 125] would guarantee good results on Sleep-EDF while on HAR, the selection range is [5 , 25] .
The choice of β does affect the final performance, but it is not tricky to search for a β for goodperformance.
We also find that the accuracy score first increases then decreases with an increasingvalue of β .
The reason might be that a large β will negatively affect the fitness loss.
D.7 Statistical Testing and Running Time Comparison
In this section, we conduct T-test on the result in Section 4.2.1 and calculate the p-values in theparenthesis of Table 11 and Table 12 (the experimental results are copied from Table 2).
Commonly, [Table] [Table] a p-value smaller than 0.05 would be considered as significant.
We can see that our model showsignificant performance gain over all baselines on all tasks.
We have also reported the running time per sweep/epoch in the tables.
When recording the runningtime, we duplicated the environment mentioned in Section 4.1, stopped other programs and ran allthe models one by one on GPUs.
We record the first 8 sweeps/epochs of all models and drop the first3 sweeps (since they might be unstable).
The average running time of the last 5 epochs are reportedin Table 11 and Table 12 while the accuracy results are from Table 2.
Note that on HAR, PTB-XLand Sleep-EDF, all methods use 128 as the batch size and on MGH, all methods use 512 as the batchsize.
The tensor based methods all use R = 32 as the rank.
We can conclude that the tensor basedmethods are generally more time-efficient than the deep learning methods with fewer parameters.
Since our model ATD and the variant ATD ss − use the augmented tensors, they cost more compared toother tensor based methods (since the size of training tensors doubles), however, we also observeempirically that they can converge faster with around half number of the epochs.
D.8 Comparison with Supervised Tensor Learning
In this subsection, we compare our model with two supervised tensor learning baselines, UMLDA Luet al.
and supervised tensor learning (STL) Tao et al.
UMLDA extracts uncorrelateddiscriminative features by sequential tensor-to-vector projections, and it includes two stages: in thefirst stage (supervised pre-training stage), it uses label information to maximize the Fisher’s discrimination criterion (FDC) as the objective and extract uncorrelated features (i.e., representations); in thesecond stage (supervised learning stage), it uses another supervised model to map the representationsto the labels.
To make a fair comparison, we use 32 as the uncorrelated feature dimension (thus, ithas the same number of learnable parameters as our model) and also use logistic regression (LR)for the second stage.
STL is an end-to-end supervised tensor learning baseline, and it is originallyproposed for binary classification with a rank-one parameterized tensor (i.e., outer product of multiplevectors).
In the comparison, we extend STL for mult-class classification by including more rank-one
[Table]
parameterized tensors (one for each class).
We use cross entropy loss to optimize the revised STLmodel.
Both baselines are implemented with PyTorch and use the training and test set only.
We havecarefully turned the baseline models to achieve higher accuracy. Result Analysis. The results are shown in Table 13. We can conclude that our methods outperformthese two supervised tensor learning baselines significantly.
The performance gap between UMLDAand our model can be explained by (i) UMLDA uses FDC criterion to design the loss function andalso forces the learned feature dimensions to be uncorrelated, which might discard some essentialclass-dependent information; (ii) our model utilizes a large set of unlabeled data. STL gives poorperformance because it is essentially a multilinear method with much fewer parameters than UMLDAand our ATD , which hurts the expressive. E Derivation of Two-sided Bound We recall the definition of L ss and L Θ ss from Section 3.1.
[Equation]
and we want to prove that
[Equation]
where r p is the label rate of class- p .
Proof.
We start by arranging L ss ,
[Equation]
[Equation]
[Equation]
From Eqn.
(16) to Eqn. (17), we use the fact that “ E [ · ] means the expectation is taken over fourinterdependent random variables, i.e., p, q, X p , Y q ", which is mentioned in Section 3.1. From Eqn.
(18) to Eqn.
(19), we use the fact that given p , the similarity of random pairs is smaller than thesimilarity of positive pairs E q, X p , Y q [ sim ( f ( X p ) , f ( Y q ))] ≤ E q, X p , Y q [ sim ( f ( X p ) , f ( Y q )) | p = q ] .
The upper bound is derived by replacing λr p 1 − r p + 1 , ∀ p with C 2 = 1 + min p
λr
p
1 − r p .
Similarly, wecan also derive the other side (lower bound) by using C 1 = 1 + max p λr p 1 − r p , which eventually gives
[Equation]