ATD : Augmenting CP Tensor Decomposition by Self Supervision
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Tensor decompositions are powerful tools for dimensionality reduction and featureinterpretation of multidimensional data such as signals.
Existing tensor decomposition objectives (e.g., Frobenius norm) are designed for ﬁtting raw data understatistical assumptions, which may not align with downstream classiﬁcation tasks.
In practice, raw input tensor can contain irrelevant information while data augmentation techniques may be used to smooth out class-irrelevant noise in samples.
Thispaper addresses the above challenges by proposing augmented tensor decomposition ( ATD ), which effectively incorporates data augmentations and self-supervisedlearning (SSL) to boost downstream classiﬁcation.
To address the non-convexityof the new augmented objective, we develop an iterative method that enables theoptimization to follow an alternating least squares (ALS) fashion.
We evaluate ourproposed ATD on multiple datasets.
It can achieve 0 .
8% ⇠ 2 . 5% accuracy gain overtensor-based baselines.
Also, our ATD model shows comparable or better perfor- mance (e.g., up to 15% in accuracy) over self-supervised and autoencoder baselineswhile using less than 5% of learnable parameters of these baseline models.
Introduction
Extracting unsupervised features from high-dimensional data is essential in various scenarios, suchas physiological signals (Cong et al., 2015), hyperspectral images (Wang et al., 2017) and fMRI(Hamdi et al., 2018).
Tensor decomposition models are often used for high-order feature extraction(Sidiropoulos et al., 2017), among these, CANDECOMP/PARAFAC (CP) decomposition is one ofthe most popular models.
The low-rank CP tensor decompositions (Kolda and Bader, 2009) assumethat the input data is composited by a small set of components, while the reduced features are thecoefﬁcients that quantify the importance of each basis, which provides a compact representation.
Under this low-rank assumption, existing tensor decomposition objectives aim to ﬁt individual datasamples with statistical error measures (Hong et al., 2020; Singh et al., 2021), e.g., Frobenius normor KL-divergence.
Though ﬁtness is an essential principle for feature reduction, common objectivefunctions do not account for downstream tasks, e.g., classiﬁcation.
Contrastive self-supervised learning (SSL) (He et al., 2020) is recently popular for unsupervisedfeature learning, which utilizes the class-preserving data augmentations (Dao et al., 2019)
andlearns an encoder that can ﬁlter out class-irrelevant information.
The optimization goal is to enforcealignments (Chen et al., 2020; Wang and Isola, 2020), ensuring that the anchor sample is closer to thepositive sample (which has the same latent class as the anchor sample) than to the negative sample(which is in a different latent class) in the embedding space.
In an unsupervised setting, positivesamples are given by data augmentations, while negative samples are hard to acquire (Arora et al.
He et al., 2020; Chen et al., 2020).
Also, previous models are mostly built on deep neuralnetworks, which are often black-box models with tens of thousands of learnable parameters.
This paper aims to incorporate both the ﬁtness and alignment principles into CP tensor decomposition 1by augmenting the common ﬁtness objective with a new self-supervised loss.
The new self-supervisedloss is based on the unbiased estimation of negative samples Chuang et al.
(2020), which effectivelyprevents the sampling bias issue (Arora et al., 2019).
The purpose of our design (i.e., introducing
SSL into CP tensor decomposition) is to learn class-aware tensor decomposition results for boostingthe downstream tensor sample classiﬁcations.
To address the non-convex subproblems from the newobjective, we formulate an iterative method, which solves least squares optimization in each step withclosed-form solution.
Main contributions are summarized below.
• We propose augmented tensor decomposition , named ATD , which learns an unsupervised CP
structure decomposition by extending the original ﬁtness objective with a self-supervised loss onthe contrastiveness of similar and dissimilar tensor samples.
• We develop an iterative method to address the non-convex subproblem from the new objective,which enables our algorithm to follow an alternative least squares fashion .
Our algorithm has
asymptotically the same complexity of each optimization sweep as the common CP-ALS.• We provide extensive evaluations on four real-world datasets and compare to recent tensor
decomposition models, autoencoder models, and self-supervised models.
Our method shows betteror comparable prediction performance in various downstream classiﬁcations while only requiring
much fewer (e
g., less than 5% of) parameters than that of deep learning baselines.
Background
Notation.
We use plain letters for scalars, such as x or X , boldface uppercase letters for matrices,e.g., X , boldface lowercase letters for vectors, e.g., x , and Euler script letters for tensors, randomvariables of tensors, and probability distributions, e.g., X .
Tensors are multidimensional arraysindexed by three or more indices (modes).
For example, an N -mode tensor X is an N -dimensionalarray of size I 1 ⇥ · · · ⇥ I N , where x i 1 ,...,i N is the element at the ( i 1 , · · · , i N ) -th position.
For matrix
X , the r -th row and column are x ( r ) and x r respectively, while x ij is for the ( i, j ) -th element.
k X k Fis the Frobenius norm.
For vector x , the r -th element is x r , and we use k x k 2 to denote the vector2-norm, h · , · i for the vector inner product,  for the outer product, and
J ·
K for the Kruskal product.
Indices in the paper start from 1 , e.g., x 1 is the ﬁrst column of the matrix.
2.1 Tensor Modeling
This paper aims to learn tensor bases from unlabeled data and then use the bases to build a featureextractor for downstream classiﬁcation.
Without loss of generality (w.r.t.
tensor order), we considerthe fourth-order tensor, e.g., a collection of multi-channel Electroencephalography (EEG) signals,
[Equation]
where T ( n ) 2 R I ⇥ J ⇥ K .
The ﬁrst dimension of T corresponds to data samples (e.g., one for eachpatient), while the other three are feature dimensions (e.g., channel by frequency by timestamp ).
Data Model.
To model the above tensor, previous works (Kolda and Bader, 2009) assume that
• There are a set of rank-one tensor components
E
= {E 1 , . . .
, E R } , which are learnable;•
The tensor data sample/slice T ( n ) admits a low-rank structure and can be represented as a weightedsum of these tensor components E , where x ( n ) denotes its coefﬁcient vector;• On top of the low-rank structure, each data sample T ( n ) also contains additional element-wise i.i.d.
Gaussian noise due to real-world distortion (e.
g., physical noise in signal measurements).
In the context of downstream classiﬁcations, we further assume that each sample T ( n ) is semanticallyassociated to one of the latent classes p 2 { 1 , . . .
, P } , and we let D p be the sample distribution of 78 class- p .
Thus, the data sample can be formulated as,
[Equation] [Equation]
where
[Equation]
✏
[Equation]
, where  is generally small .
Here a r , b r , c r are the learnable parameters, which produces the rank-one component E r , and theyare the column vectors of { A 2 R I ⇥ R , B 2 R J ⇥ R , C 2 R K ⇥ R } , referred as "bases".
CANDECOMP/PARAFAC Decomposition (CPD).
Given the above tensor model, standard CPDonly captures the i.i.d.
Gaussian noise by minimizing the negative log-likelihood (NLL), whichresults in the following standard ﬁtness/reconstruction loss,
[Equation]
Here, the Kruskal product
J ·
K outputs a fourth-order reconstructed tensor from four input factormatrices.
For consistency, if the ﬁrst input is a vector, the output is considered as a third-order tensor.
2.2 Problem Formulation
CP decomposition seeks a low-rank reconstruction, without special consideration for the downstreamtask.
In this paper, we are motivated to improve the CPD model by exploiting the latent classes (in anunsupervised way) and learn good bases to provide better class-aware features for classiﬁcation.
What are Good Bases?
This paper considers two design principles for good bases.
The ﬁrstprinciple is ﬁtness , which requires a low-rank tensor reconstruction with the bases.
Second, datasamples associated with the same latent class should be decomposed into similar coefﬁcient vectors,with the bases, while the vectors should be dissimilar if the samples are from different latent classes.
This principle is called alignment , which is important for classiﬁcation but not considered in thestandard tensor decomposition.
In this paper, we assess the quality of the learned bases by the
performance of downstream classiﬁcation, where the coefﬁcient vectors (obtained using the bases viadecomposition) are the feature inputs (into the downstream classiﬁer).
Working Pipelines.
To put it succinctly, the paper tackles an unsupervised learning problem whileusing downstream supervised classiﬁcation for evaluation.
The procedures are brieﬂy outlined:
• First, we learn the bases { A , B , C } from a large set of unlabeled data.
The loss function isdeveloped in consideration of the ﬁtness and alignment (deﬁned in the next section)
principles.•
Then, we construct the following feature extractor given { A , B , C } .
The feature vector of anew sample is obtained by the closed-form solution of the least squares problem ( ↵ > 0 is ahyperparameter),
[Equation]
Note that, when f ( · ) is applied to a batch of samples, it outputs a coefﬁcient matrix.
• Next, we evaluate the feature extractor with a set of labeled data.
Given f ( · ) , we ﬁrst apply it onthe labeled data to extract their features and then train an additional logistic regression model (as
the downstream classiﬁer) on top of the extracted features, so that the result of classiﬁcations will
implicitly reﬂect how good the bases are.
Augmented Tensor Decomposition
We show our model in Figure 1.
The design is inspired by the recent popularity of SSL.
To exploitthe latent class information, we introduce class-preserving data augmentation into CPD model anddesign self-supervised loss to constrain the learned low-rank features (i.e., the coefﬁcient vectors).
Data Augmentation 2 .
In general, data augmentation methods are chosen to perturb the raw data whilepreserving class label information.
Given a sample T ( n ) , we assume that after data augmentation, itsperturbation ˜ T ( n ) preserves the label and admits the component-based representation as in Eqn.
[Figure]
The design of our self supervised loss corresponds to the alignment principle, which is based onpairwise feature similarity and dissimilarity.
We call a pair of data samples from the same latentclass as positive pair , a pair of samples from different latent classes as negative pair and a pair ofindependent samples (two random samples from the dataset) as random pair .
Intuitively, an anchorplus a positive sample composes a positive pair, similarly for negative pairs and random pairs.
In thiswork, our self-supervised loss aims to maximize the feature similarity between positive pairs andminimizes that between negative pairs in an unsupervised way (no labels during optimization).
Formally, let X p , Y p be discrete random variables (of tensor samples) distributed as D p , p 2{ 1 , . .
. , P } .
We want to minimize the following objective when no class labels are given ,
[Equation]
where   1 is a hyperparameter and
[Equation]
Here L pos maximizes the feature similarity of positive pairs while L neg minimizes the feature 130 similarity of negative pairs.
f ( · ) is the feature extractor, deﬁned in Eqn.
(2), and the similaritymeasure is given by cosine distance, parameterized by two random variables,
[Equation]
To this end, the key is to implement the self-supervised loss, i.e., Eqn.
(3), in an unsupervised setting.
Speciﬁcally, we want to construct the sampler of positive pairs and the sampler of negative pairswith unlabeled data only.
The sampler of positive pairs (in short, positive sampler) can be easilyapproximated by data augmentation techniques, which provides "surrogate" positive pairs (given anysample as the anchor, we apply data augmentation methods to generate a perturbed data as positivesample, and then the anchor plus the perturbed data is a positive pair).
However, the negative sampleris infeasible, without labels.
As a compromise, previous works He et al.
; Chen et al.
(2020)consider using random sampler to replace the negative sampler given that the random sampler canbe easily achieved by picking two independent samples from the dataset.
However, this practice isshown to induce sampling bias and hurts the learned representation Arora et al.
(2019); Chuang et al.(2020) since a random pair may be from the same latent class.
Construction of Negative Sampler.
In this paper, we consider using the law of total probabilityto construct the negative sampler in a statistical way.
Formally, assume r p is the label rate of latent
class- p (thus, we have
P p r p = 1 ), we apply the law of total probability and the following holds,
[Equation]
Here, the usage of E [ · ] means that the expectation is taken over four interdependent random variables,i.e., p, q, X p , Y q , while E p,q [ · ] means that p, q is ﬁxed and thus it is only taken over two randomvariables, i.e., X p , Y q .
The result shows that the negative sampler can be equivalently replaced by aweighted combination of the random sampler and positive sampler.
Here we do not have access tor p , 8 p with unlabeled data, this issue is dealt with later.
Self-supervised Loss.
Consequently, we can reformulate our self-supervised loss as,
[Equation]
From Eqn.
(5) to Eqn.
(6), we use the results in Eqn.
Two-sided Bound.
The above form still requires label rate information, i.e., r p , 8 p , we thereforeconsider using the following approximation to the above loss L ss ,
[Equation]
Here,   0 is a hyperparameter, while L ss can be bounded as (derivations in appendix E),
[Equation]
The equivalence is established when C 1 = C 2 , i.e., the class labels are balanced.
To simplify thederivation, we ignore  in the following and let  be a new hyperparameter.
Also, the constants
C 1 and C 2 can be absorbed into a weight hyperparameter  , given in the next section.
This boundimplies that, an easy-to-compute  L ⇥ ss (  ) is often a good approximation of L ss for some  .
The nextsection speciﬁes how to compute  L ⇥ ss (  ) unsupervisedly as our empirical self-supervised loss .
3.2 The Objective of ATD
Empirical Estimator.
We obtain an empirical estimator of L ⇥ ss with Monte Carlo method.
Suppose
T and ˜ T are the input tensor and the augmented tensor respectively, and X = f ( T ) , ˜ X
= f ( ˜ T )
R N ⇥ R are the coefﬁcient/feature matrices.
We use the row vectors of X , ˜ X to estimate Eqn.
The ﬁrst term E [ sim ( f ( X p ) , f ( Y q ))] essentially requires a random sampler, which is approximatedby the average cosine similarity of all possible pairs of non-corresponding row vectors of X , ˜ X , whilethe second term E [ sim ( f ( X p ) , f ( Y q ))
| p = q ] requires a positive sampler, which is estimated bythe average cosine similarity of all pairs of corresponding row vectors,
[Equation]
where D ( X ) = diag ⇣ k
[Equation]
k
[Equation]
, · · · , k x
[Equation]
N
[Equation]
k⌘ is the row-wise scaling matrix and
G
[Equation]
1 N  +1 N ( N  1) · · ·  +1 N ( N  1)  +
N ( N  1)  1 N · · ·  +1 N ( N  1) · · · · · · · · · · · ·  +1 N ( N  1)  +1 N ( N  1) · · ·  1 N  .
Note that, the form in Eqn.
(9) is signiﬁcantly different from tensor discriminant analysis (Jia et al.
Tao et al., 2007), which integrates the actual label information as a regularizer and is alsodifferent from graph regularized tensor decomposition (Cai et al., 2010), which incorporates sideinformation, such as geometrical positions (Maki et al., 2018).
Compared to standard noise contrastiveestimation (NCE) (Gutmann and Hyvärinen, 2010; Chen et al., 2020) in the area of contrastive SSL,our SSL form in Eqn.
(9) considers a subtraction form instead of the softmax formulation, making itamenable to quadratic optimization, as we will show in Sec. 3.3.
Overall Objective.
According to Eqn.
(8), the self supervised loss L ss is bounded by L ⇥ ss (  ) , whilethe constants (i.e., C 1 , C 2 ) can be absorbed into a weight hyperparameter  .
We let the empirical selfsupervised loss, ˜ L ss =
˜
L ⇥ ss (  ) .
Our objective follows both the ﬁtness (i.e., CPD reconstruction loss)and alignment (i.e., self-supervised loss) principles, while also considering Tikhonov regularization(Golub and Von Matt, 1997) to constrain the scale of all parameters,
L
[Equation]
10) where
[Equation]
The objective has (i) three hyperparameters,  , ↵ ,  > 0 ; (ii) ﬁve factor matrices, A , B , C , X , ˜ X .
3.3 Alternating Least Squares Optimization
Ideally, we would like to use alternative least squares (ALS) algorithm (Sidiropoulos et al., 2017)for optimizing the objective, where each factor matrix is updated in a sequence by solving leastsquares subproblems.
With large scale tensors, we can also resort to mini-batch stochastic ALS (Caoet al., 2020) to reduce memory footprint of the decomposition.
However, the objective in Eqn.
(11)is non-convex to X and ˜ X , respectively.
For addressing the non-convex subproblem, we design aniterative method in this section, which only involves solving least square problems.
Addressing Non-convex Subproblem.
We use the subproblem of X as an example.
Given A , B ,
C , ˜ X ﬁxed, we want to minimize Eqn.
(10) by ﬁnding the optimal solution, denotd as X ⇤ ,
[Equation]
• First, we are interested to ﬁnd that the matrix-formed problem in Eqn.
(12) can be
decomposed intorow-wise subproblems and to obtain the solution of Eqn.
(12), it is sufﬁce to solve each subproblemindependently.
Let us consider the subproblem of the k -th row, which is
[Equation]
where T ( k ) is the k -th slice of T , and g ( k ) is the k -th row of G (  ) .•
Here, Eqn.
(13) is still non-convex.
We let the derivative of Eqn.
(13) objective to be zero andarrange the terms, which yields,
[Equation]
where
[Equation]
Here x , v 1 , v 2 are row vectors and V 3 is a matrix.
T ( k ) 1 is the 1 -mode unfolding of T ( k ) ,  is the
Khatri-Rao product and ⇤ is the Hadamard product (i.e., element-wise product).•
We consider the following iterative rule and the ﬁxed point is a solution for Eqn.
(14), which is astationary point of Eqn.
[Equation]
We use an initial guess x init (obtained by solving Eqn.
(13) with while  = 0 , which is a leastsquare problem) to start
and then we repeat Eqn.
(15) for each row (i.e., each k ) and let the improvedguess be the initial guess, x init x impr , to iteratively improve the result.
Theorem 1 (proof in appendix A) ensures that Eqn.
(15) converges linearly if  is chosen to besufﬁciently small.
In appendix B, we verify the liner convergence and also empirically show that oneround of Eqn.
(15) is sufﬁcient in our experiment, where  = 2 .
The non-convex subproblem of ˜ Xcan be solved in the same way.
Theorem 1.
Given non-zero row vectors v 1 , v 2 , u 0 2 R d , non-singular matrix V 3 2 R d ⇥ d and > 0 .
The sequence { u t } , generated by u t +1 = v 1 V 3   v 2 2 k
[Equation]
k
[Equation]
[Equation]
[Equation]
where u ⇤ is the ﬁxed point and m =
min t k u t k 2 , M = max t k u t k 2 are the bound of the sequence.
With a good u 0 and a sufﬁciently small  , we can safely assume
< m  M < 1 .
Optimization Procedures.
To minimize Eqn.
(11), we alternatively update A , B , C , X , and ˜ X ,where each subproblem involves only solving least squares problems with closed-form solutions.
With large-scale tensors (as in the experiments), we optimizes the factors in mini-batches.
Betweenmini-batches, the basis factors A , B , C are shared and updated incrementally.
We summarize thealgortihms in appendix C. The computation head of the algorithm is matricized tensor times KhatriRao product (MTTKRP).
The complexity of our optimization algorithm is asymptotically the sameas applying CP-ALS, which costs O ( NIJKR ) to sweep over the whole tensor once.
Experiments
This section presents the experimental evaluations.
Due to space limitation, additional details,including data augmentations and baseline implementation, are presented in appendix D.
4.1 Experimental Setup
Data Preparation.
We use four real-world datasets: (i) Sleep-EDF
(Kemp et al., 2000), whichcontains EOG, EMG and EEG Polysomnography recordings; (ii) human activity recognition (HAR)(Anguita et al., 2013) with smartphone accelerometer and gyroscope data; (iii) Physikalisch Technische Bundesanstalt large scale cardiology database (PTB-XL)
(Alday et al., 2020) with 12-lead ECGsignals
; (iv) Massachusetts General Hospital (MGH) (Biswal et al., 2018) datasets with multi-channel
EEG waves.
All datasets are split into three disjoint sets (i.e., unlabeled, training and test) by subjects,while training and test sets have labels.
Basic statistics are shown in Table 1.
All models (baselinesand our ATD ) use the same augmentation techniques: (a) jittering, (b) bandpass ﬁltering, and (c) 3Dposition rotation.
We provide an ablation study on the augmentation methods in appendix D.5.
Baseline Methods.
We include the following comparison models from different perspectives:
• Tensor based models : ATD ss  is our variant, which removes the self-supervised loss from the
objective in Eqn.
(10); Stochastic alternating least squares (SALS) applies on the the CPD objectivewith Tikhonov regularizer, which works on large tensors; Graph regularized SALS (GR-SALS)augments the objective of SALS with a graph regularizer (Maki et al., 2018;
Cai et al., 2010),deﬁne as Tr
[Equation]
[Table] [Table]
• Self-supervised models : SimCLR - r
(Chen et al., 2020) and BYOL - r (Grill et al., 2020) are twopopular SSL models with their own objective functions, where r indicates the size of the outputrepresentation.• Auto-encoder models : AE - r denotes a CNN based autoencoder with mean square error (MSE)reconstruction loss, and AE ss - r denotes the same autoencoder model with standard NCE loss inthe bottleneck layer, where r is the representation size.
All models use the unlabeled set to train a feature encoder and use training and test sets to evaluate.
Note that, deep neural network models use the same CNN backbone.
In appendix D.8, we have alsocompared with two recent supervised tensor learning models, which shows the usefulness of our ATDand the large unlabeled set, especially in low-label rate scenarios.
Evaluation and Environments.
We evaluate model performance mainly based on classiﬁcationaccuracy , where we train an additional logistic classiﬁer (He et al., 2020) on top of the feature encoder.
Also, for different models, we compare their number of learnable parameters .
The experiments areimplemented by Python 3.8.5, Torch 1.8.0+cu111 on a Linux workstation with 256 GB memory, 32core CPUs (3.70 GHz, 128 MB cache), two RTX 3090 GPUs (24 GB memory each).
All trainingis performed on the GPU.
For tensor based models, we use R = 32 and implement the pipeline in
CUDA manually, instead of using torch-autograd .
4.2 Experimental Results
This section shows the experimental results on downstream classiﬁcation.
We use all the unlabeleddata to train the encoder or feature extractor, and use training data (since Sleep-EDF and MGHdatasets have enough training samples, we randomly selected a subset of them) for learning adownstream classiﬁer and use all test data.
Each experiment is conducted with ﬁve different randomseeds and the mean and standard deviations are reported.
The metrics are the accuracy and thenumber of learnable parameters .
All models have 32-dim features in the end, except that for twoself-supervised baselines and autoencoder models, which have 128-dim options.
4.2.1 Better Classiﬁcation Accuracy with Fewer Parameters
From Table 2, ATD shows comparable or better performance over the baselines.
We have also reportedthe running time per epoch/sweep in appendix D.7 for all models.
Compared to the variant ATD ss  ,our ATD can improve the accuracy by 1 . 0% ⇠ 1 . 9% , which shows the beneﬁt of the inclusion ofself-supervised loss.
SALS and ATD ss  have similar performance, while their objectives differ in that ATD ss  considers the Frobenius norm of the augmented data.
Thus, their accuracy gap is caused bythe use of data augmentation.
Also, the experiments show that the ﬁtness and alignment principlesare both important.
We observe that with a self-supervised loss (i.e., alignment ), AE ss can givesigniﬁcant improvement over AE, while ATD shows ⇠ 8% accuracy gain over the self-supervisedmodels on MGH dataset, since we can better preserve the data with a reconstruction loss (i.e., ﬁtness ).
Moreover, the table shows that tensor based models require fewer parameters, i.e., less than 5%of parameters compared to deep learning models.
On HAR, the deep unsupervised models showpoor performance due to (i) they may not optimize a large number of parameters on middle-scaledataset; (ii) movement signals in HAR might have few degrees of freedom, which matches wellwith the low-rank assumption of tensor methods.
On large-scale Sleep-EDF, self-supervised modelsoutperforms ATD marginally since they have more parameters thus can capture more information.
4.2.2 Better Performance in Low-label Rate Scenarios
On the MGH dataset, we also show the effect of varying the amount of training data in Figure 2.
Weinclude an end-to-end convolutional neural network (CNN) model based on (Biswal et al., 2018),called Reference CNN , which is a supervised model and only uses the training and test sets.
To bemore readable, we separate the comparison ﬁgure into two sub-ﬁgures: the left compares our ATDmodel with self-supervised and auto-encoder baselines and the right one compares ATD with tensorbaselines and the reference model, and the scale of y-axis on two sub-ﬁgures are the same.
We ﬁnd that all unsupervised models outperform the supervised reference CNN model in scenarioswith fewer training samples.
With more training data, the performance of all models get improved,especially the reference CNN model, which can optimize the encoder and predictive layers in anend-to-end way and ﬁnally outperforms our ATD when more training samples is available.
4.2.3 Stable Results with Hyperparameter Variation
For a comprehensive evaluation, we also conduct ablation studies on the effect of the data augmentation methods and on hyperparameters.
Due to space limitation, we move the experimental settings andresults to appendix D.6, while summarizing the general conclusions here: (i) with more diverse dataaugmentation methods, the ﬁnal results are relatively better; (ii) with a larger rank R , the performancewill be better generally; (iii) our ATD is not sensitive hyperparameters ↵ ,  ,  .
Conclusion
This paper introduces the concept of self-supervised learning for tensors and proposes Augmented
Tensor Decomposition ( ATD ) and the ALS-based optimization.
We show that by explicitly contrastingpositive and negative samples, the decomposition results are more aligned with downstream classiﬁcation.
On four real-world datasets, we show the advantages of our model over various unsupervisedmodels and with fewer training data, our model even outperforms the reference supervised models.
Compared to deep learning methods, tensor based models are not as ﬂexible in processing multimodaland diverse inputs, such as natural images.
However, applying tensor decomposition on the outputsof earlier layers of pre-trained deep neural networks may be a feasible way to address the weaknesses.
This direction would be interesting for future work.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A] Please do not modify the questions and only use the provided macros for your answers.
Note that the Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Conclusion
(c) Did you discuss any potential negative societal impacts of your work?
[No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] see Theorem 1(b)
Did you include complete proofs
of all theoretical results?
[Yes] See appendix A
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] See
supplementary (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See appendix D
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Table
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes](c)
Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[Yes] See appendix D.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?
[N/A]5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]