A NEW FRAMEWORK FOR TENSOR PCA BASED ON TRACE INVARIANTS
Anonymous authors
Paper under double-blind review
A BSTRACT
We consider the Principal Component Analysis (PCA) problem for tensors T ∈
( R n ) ⊗ k of large dimension n and of arbitrary order k ≥ 3 .
It consists in recovering a spike v ⊗ k(related to a signal vector v 0 ∈ R n ) corrupted by a Gaussian noise tensor Z ∈ ( R n ) ⊗ k such that T
= β v ⊗ k
+ Z where β is the signal-to-noise ratio.
In this paper, we propose a new framework based on tools developed by the theoretical physics community to address this important problem.
They consist in trace invariants of tensors built by judicious contractions (extension of matrix product) of the indices of the tensor T .
Inspired by these tools, we introduce a new process that builds for each invariant a matrix whose top eigenvector is correlated to the signal for β sufﬁciently large.
Then, we give examples of classes of invariants for which we demonstrate that this correlation happens above the best algorithmic threshold ( β ≥ n k/ 4 ) known so far.
This method has many algorithmic advantages: (i) it provides a detection algorithm linear in time and with only O (1) memory requirements (ii) the algorithms are very suitable for parallel architectures and have a lot of potential of optimization given the simplicity of the mathematical tools involved (iii) experimental results show an improvement of the state of the art for the symmetric tensor PCA.
Furthermore, this framework allows more general applications by being able to theoretically study the recovery of a spike in the form of v 1 ⊗ · · · ⊗ v k with different dimensions ( T ∈ R n 1 × n 2 ×···× n k with n 1 , . . . , n k ∈ N ) as well as the recovery of a sum of different orthogonal spikes.
We provide experimental results to these different cases that match well with our theoretical ﬁndings.
I NTRODUCTION
Powerful computers and acquisition devices have made it possible to capture and store real-world multidimensional data.
For practical applications (Kolda & Bader (2009)), analyzing and organizing these high dimensional arrays (formally called tensors) lead to the well known curse of dimensionality.
Thus, dimensionality reduction is frequently employed to transform a high-dimensional data set by projecting it into a lower dimensional space while retaining most of the information and underlying structure.
One of these techniques is Principal Component Analysis.
PCA has made remarkable progress in a large number of areas thanks to its simplicity and adaptability (Jolliffe &
Cadima (2016)).
In the Tensor PCA, as introduced by Richard & Montanari (2014), we consider a simple model where we attempt to detect and retrieve an unknown unit vector v 0 from noise-corrupted multilinear measurements put in the form of a tensor T .
Using the notations found below, our model consists in:
[Equation]
with Z a pure Gaussian noise tensor of order k and dimension n with identically independent distributed (iid) standard Gaussian entries:
Z i 1 ,i 2 ,...,i k ∼ N (0 , 1) and β the signal-to-noise ratio.
To solve this important problem, many methods have been proposed.
However, practical applications require optimizable and parallelizable algorithms that are able to avoid the high computationally cost due to an unsatisfactory scalability of some of these methods.
A summary of the time and space requirement of some existent methods can be found in Anandkumar et al.
One way to achieve this parallelizable algorithm is through methods based on tensor contractions (Kim et al.
(2018)) which are extensions of the matrix product.
These last years, tools based on tensor contractions have been developed by theoretical physicists where random tensors have emerged as a generalization of random matrices.
We want to investigate the algorithmic threshold of tensor
PCA and some of its variants using the theoretical physics approach in the hope that it leads to new insights and knowledge in tensor PCA.
Tensor PCA and tensor decomposition (the recovery of multiple spikes) is motivated by the increasing number of problems in which it is crucial to exploit the tensorial structure (Sidiropoulos et al.
Recently it was successfully used to address important problems in unsupervised learning (learning latent variable models, in particular latent Dirichlet allocation Anandkumar et al.
Anandkumar et al.
(2015)), supervised learning (training of two-layer neural networks, Janzamin et al.
and reinforcement learning (Azizzadenesheli et al.
Related work Tensor PCA was introduced by Richard & Montanari (2014) where the authors suggested and analyzed different methods to recover the signal vector like matrix unfolding and power iteration.
Since then various other methods were proposed.
Hopkins et al.
(2015) introduced algorithms based on the sum of squares hierarchy with the ﬁrst proven algorithmic threshold of n k/ 4 .
However this class of algorithm generally require high computing resources and complex mathematics.
Other studied methods have been inspired by different perspectives like homotopy in Anandkumar et al.
(2017), statistical physics (Arous et al.
(2020), Ros et al.
(2019), Wein et al.
(2019) and Biroli et al.
quantum computing (Hastings (2020)) as well as statistical query (Dudeja & Hsu (2020)).
Recently, a fundamentally different set of mathematical tools that have been developed for tensors in the context of high energy physics have been used to approach the problem.
They consist in trace invariants of degree d, d ∈ N , obtained by contracting pair of indices of d copies of the tensor T .
They have been used in Evnin (2020) to study the highest eigenvalue of a real symmetric Gaussian tensor.
Subsequently, Gurau (2020) provided a theoretical study on a function based on an inﬁnite sum of these invariants.
Their results suggest a transition phase for the highest eigenvalue of a tensor for β around n 1 / 2 in a similar way to the BBP transition in the matrix case (Baik et al.
Thus, this function allows the detection of a spike.
However evaluating it involves computing an integral over a n-dimensional space, which may not be possible in a polynomial time.
The contribution of this paper is the use of these invariant tools to build tractable algorithms with polynomial complexity.
In contrast to Gurau (2020), instead of using a sum of an inﬁnite number of invariants, we select one trace invariant with convenient properties to build our algorithms.
It lets us detect the presence of the signal linearly in time and with a space requirement in O (1) .
Moreover, in order to recover the signal vector besides simply detecting it, we introduce new tools in the form of matrices associated to this speciﬁc invariant.
Within this framework, we show as particular cases, that the two simpler graphs (of degree two) are equivalent to the tensor unfolding and to homotopy algorithm (which is equivalent to average gradient descent).
These two algorithms are the main practical ones known from the point of view of space and time requirement (Anandkumar et al.
(2017) provides a table comparison).
Interestingly, our framework allows us to go further by using graphs of higher degrees and can see in that case that the tetrahedral graph (degree four) performs better in the numerical experiments on the symmetric case and gives nearly optimal results for values of n of order 100 .
We also derive an algorithmic threshold, that is for the best of our knowledge novel and new, that generalizes the threshold n k/ 4 for a spike v ⊗ k of a tensor T ∈ (cid:78) R n to the case of a spike v 1 ⊗ · · · ⊗ v k and T ∈ (cid:78) R n i where the threshold to recover the vector v i is given by max( n 2 i , ( (cid:81)
ki =1 n i )) .
Notations We use bold characters T , M , v for tensors, matrices and vectors and T ijk , M ij , v i for their components.
[ p ] denotes the set { 1 , . .
. , n } .
A real k − th order tensor is of order
k if it is a member of the tensor product of R n i , i ∈
[ k ] : T ∈ (cid:78) k
i =1 R n i .
It is symmetric if T i 1 ...
i k
=
T τ ( i 1 ) ...
τ ( i k ) ∀ τ ∈ S k where S k is the symmetric group (more details are provided in Appendix B).
For a vector v ∈ R n , we use v ⊗ p ≡ v ⊗ v ⊗ · · · ⊗ v ∈ (cid:78) p R n to denote its p -th tensor power.
(cid:104) v , w (cid:105) denotes the scalar product of v and w .
Let’s deﬁne the operator norm, which is equivalent to the highest eigenvalue of a tensor of any order: (cid:107) X (cid:107) op ≡ max { X i 1 ,...,i k ( w 1 ) i 1 . . .
( w k ) i k , ∀ i ∈ { 1 , . . . , n } , (cid:107) w i (cid:107) ≤ 1 } The trace of A is denoted Tr( A ) .
We denote the expectation of a variable X by E ( X ) and its variance by σ ( X ) .
We say that a function f is negligible compared to a positive function g
and we write f = o ( g ) if lim n →∞ f/g → 0 .
Einstein summation convention It is important to keep in mind throughout the paper that we will follow the Einstein summation convention: when an index variable appears twice in a single term and is not otherwise deﬁned, it implies summation of that term over all the values of the index.
For example: T ijk T ijk ≡ (cid:80) ijk T ijk T ijk .
It is a common convention when addressing tensor problems that helps to make the equations more comprehensible.
G ENERAL F RAMEWORK FOR S IGNAL D ETECTION AND R ECOVERY 2. W HAT DO WE USE TO STUDY THE SIGNAL ?
An important concept in problems involving matrices is the spectral theory.
It refers to the study of eigenvalues and eigenvectors of a matrix.
It is of fundamental importance in many areas.
In machine learning, the matrix PCA computes the eigenvectors and eigenvalues of the covariance matrix of the features to perform a dimensional reduction while ensuring most of the key information is maintained.
In this case, the eigenvalues is a very efﬁcient tool to describe data variability.
In the case of signal processing, eigenvalue can contain information about the intensity of the signal, while the eigenvector points out to its direction.
Lastly, a more theoretical example involves quantum physics where the spectrum of the matrix operator is used to calculate the energy levels and the state associated.
In all of these examples, an important property of the eigenvalues of a n-dimensional matrix M is its invariance under orthogonal transformations { M → OMO − 1 , O ∈ O( n ) } where O( n ) is the n -dimensional orthogonal group (i.e. the group of real matrices that satisﬁes OO (cid:62)
=
I n ).
Since these transformations essentially just rotate the basis to deﬁne the coordinate system, they must not affect intrinsic information like data variability, signal intensity or the energy of a system.
The eigenvalues are able to capture some of these inherent information, but recovering the complete general information requires computing their respective eigenvectors (for example to ﬁnd the principal component, the direction of the signal or the physical state).
There are more such invariants than eigenvalues.
Another important set worth mentioning are the traces of the n ﬁrst matrix powers
Tr( A ) , Tr (cid:0)
A 2 (cid:1) , . . . ,
Tr( A n ) .
Obtaining them uses slightly different methods than eigenvalues, but they contain the same information since each set can be inferred from the other through some basic algebraic operations.
On the basis of the matrix case, we expect that for a tensor T ∈ (cid:78)
ki
=1 R n i , tensor quantities that are invariant under orthogonal transformations ( T a 1 j ...a kj → O (1) a 1 j b 1 j . . .
O ( k ) a kj b
[Equation]
T
[Equation]
b kj for O ( i ) ∈
O( n i ) ∀ i ∈
[ k ] . ) should capture similar intrinsic information like the intensity of the signal, and conceivably, there should be other objects related to these quantities that are able to indicate the direction of the signal.
However, the concept of eigenvalue and eigenvector is ill deﬁned in the tensor case and not practical giving that the number of eigenvalues is exponential with the dimension n (Qi (2005), Cartwright & Sturmfels (2013)) and computing them is very complicated.
In contrast, we have a very convenient generalization of the traces of the power matrices for the tensors that we call trace invariants.
They have been extensively studied during the last years in the context of high energy physics and many important properties have been proven.
We will ﬁrst give a more formal deﬁnition of trace invariants.
Let T be a tensor whose entries are
T i 1 ,...,i k .
Let’s deﬁne a contraction of a pair of indices as setting them equal to each other and summing over them, as in calculating the trace of a matrix ( A ij → (cid:80) ni
=1 A ii ).
The trace invariants of the tensor T correspond to the different way to contract pairs of indices in a product of an even number of copies of T .
The degree of the trace invariants consists in the number of copies of T contracted.
For example, (cid:80) i 1 ,i 2 ,i 3 T
i 1 i 2 i 3 T i 1 i 2 i 3 and (cid:80) i 1 ,i 2 ,i 3 T i 1 i 2 i 2 T i 1 i 3 i 3 are trace invariants of degree 2.
In the rest of this paper, we will use the Einstein summation convention by dropping the sum symbol and imply summation when an index variable appears twice.
Three important examples of trace invariants are: the melon diagram (Figure 1b), the tetrahedral (1c) and the tadpole (1d).
Avohou et al.
(2019) provides a thorough study about the number of trace invariants for a given degree d .
A very useful asset of these invariants is that we are able to compute their expectation for tensors whose components are Gaussian using simple combinatorial analysis (Gurau (2017)).
[Figure]
2. W HAT DO WE USE TO RECOVER THE SIGNAL ?
As previously mentioned in Section 2.1, an invariant should be able to detect a signal.
But if our goal is to recover it, we should ﬁnd mathematical objects that are able to provide a vector.
To this effect, we introduce in this paper a new set of tools in the form of matrices.
We denote by M G ,e the matrix obtained by cutting an edge e of a graph G in two half edges (see Figure 3 for an example).
Indeed, this cut amounts to not summing over the two indices i
and i 2 associated to these two half-edges and using them to index the matrix instead.
We will drop the index G , e of the matrix when the context is clear.
[Figure]
P HASE T RANSITION WITHIN THIS F RAMEWORK
We can represent the tensor from which we hope to extract the signal represented graphically as:
[Equation]
This decomposition leads us naturally to decompose in a similar way a tensor trace
invariant I G ( T ) into two parts, separating a ﬁrst contribution I ( N ) G ( T ) associated to the pure noise tensor contribution and a second part I ( S ) G ( T ) enclosing all the other contributions, which are resulting from the addition of the signal.
[Equation]
An identical decomposition can be carried out for the matrix.
Let’s consider a tensor T , a graph G and its associated trace invariant I G ( T ) .
Let’s denote I (cid:48)G ( T ) the invariant associated to the subgraph obtained by removing from G the edge e and its two vertices.
We can distinguish three kind of contributions to the matrix M G ,e that we denote M ( N ) G ,e , M ( D ) G ,e and M ( R ) G ,e , illustrated in Figure(where we denoted the invariant I (cid:48)G ( T ) by I (cid:48) and dropped the index G , e for simplicity). Lemma 1. E [Equation]
Using the lemma 1, we identify three possible phases depending on which matrix operational norm is much larger than the others:
• No detection and no recovery: If (cid:13)(cid:13) M ( N ) − E ( M ( N ) ) (cid:13)(cid:13) op (cid:29) (cid:13)(cid:13) M ( D ) (cid:13)(cid:13) op , (cid:13)(cid:13) M ( R ) (cid:13)(cid:13) op
then no recovery and no detection is possible we can’t distinguish if there is a signal.
It is for example the phase for β → 0 .
I (cid:48)
I (cid:48)
( N )
I (cid:48) ( S ) I (cid:48)
I (cid:48) I (cid:48)
i 1 i... i 1 i...
i 1 i... i 1 i...
i 1 i... i 1 i...
= =
Z i 1 ..
Z i 2 .. ...
+ +
Z i 1 ...
Z i 2 ... ...
+ +
( v i 1 Z i 2 .. ...
+ + v i
Z i 1 .. ... )
+ + v i 1 v i 2 ...
M i 1 i= =
M ( N ) i 1 i+ +
M ( D ) i 1 i+ +
M ( R ) i 1 i
[Figure]
Signal contribution to the detection
Signal contribution to the recovery
[Equation]
• Detection but no recovery: If (cid:13)(cid:13) M ( D ) (cid:13)(cid:13)
op (cid:29) (cid:13)(cid:13)
M
[Equation]
N [Equation]
N
[Equation] M [Equation] R ) (cid:13)(cid:13) op then detection but no recovery.
We can detect the presence of the signal (thanks to the highest eigenvalue) but we can’t recover the signal vector since the leading eigenvector is not correlated to the signal vector.
• Detection and recovery: (cid:13)(cid:13)
M ( R ) (cid:13)(cid:13) op (cid:29) (cid:13)(cid:13)
M ( N ) − E ( M ( N ) )
(cid:13)(cid:13) op , (cid:13)(cid:13) M
( D ) (cid:13)(cid:13) op .
We recover the signal vector.
It is for example the phase for β → ∞ .
2. A LGORITHMIC T HRESHOLD FOR A G ENERAL G RAPH
We can now state the important algorithms that will be essential for this paper.
It is important to keep in mind that the following claims concern the large n limit.
Empirically, the approximation of large n limit seems valid for n > 25 .
Input: The tensor T = β v ⊗ k + Z
Goal: Detection of v 1 Calculate the scalar α =
I G ( T ) ; 2 Compare | α − E
( I G ( B )) | with the variance σ ( I G ( B )) for a distribution of B that follows a
Gaussian random tensor model.
Result: Gives the probability of the presence of a spike
The ﬁrst algorithm gives a criteria for distinguishing a pure noise tensor from a tensor with a spike.
Denoting the expectation and the variance of the trace invariant associated to a graph G for a Gaussian random tensor model (from which we sample random tensors ( B ) ) by E ( I G ( B )) and σ ( I G ( B )) .
The algorithm consists simply in calculating the trace invariant of the tensor and comparing its distance from E ( I G ( B )) with σ ( I G ( B )) .
It is straightforward to see that calculating a trace invariant (which is a scalar) like T ijk T ijk only needs O (1) memory.
Theorem 2.
Let G be a graph of degree d , ∃ β det > 0 so that Algorithm 1 detects the presence of a signal for β ≥ β det .
The second algorithm is able to recover the spike in a tensor T through the construction of the matrix of size n × n M G ,e ( T ) associated to a given graph G and edge e .
Algorithm 2: Recovery algorithm associated to the graph G and edge e
Input: The tensor T = β v ⊗ k
+ Z
Goal: Estimate v
Result: Obtaining an estimated vector v
Theorem 3.
Let G be a graph of degree d , ∃ β rec > 0 so that Algorithm 2 gives an estimator v so that v is strongly correlated to v 0 ( (cid:104) v , v 0 (cid:105) > 0 . 9 ) for β ≥ β rec .
The last theorem gives a lower bound to the threshold above which we can detect and recover a spike using a single graph.
Interestingly, this threshold which appears naturally in our framework, matches the threshold below which there is no known algorithm that is able to recover the spike in polynomial time.
Theorem 4.
Let k ≥ 3 .
It is impossible to detect or recover the signal using a single graph below the threshold β ≤ n k/ 4 which is the minimal variance of any graph.
S OME APPLICATIONS OF THIS FRAMEWORK
Using these algorithms, we are now able to investigate the performance of our framework in various theoretical settings.
In the ﬁrst two subsections, we study the algorithms associated to two trace invariants of degree 2.
They consist of the melonic diagram, which gives a very practical detection algorithm, and whose recovery algorithm is a variant of the unfolding algorithm, and the tadpole diagram whose recovery algorithm is similar to the homotopy algorithm.
The third subsection goes further in terms of graph’s degree and investigate the algorithms associated to the perfect one-factorization graphs (consisting in the tetrahedral when k = 3 ).
The last two sections are an illustration of the versatility of this framework.
We study the case the dimensions n i of the tensor T ( T ∈ (cid:78) ki
=1 R n i ) are not necessarily equal, which is important for practical applications where the dimensions are naturally asymmetric.
Our methods allows us to derive a new algorithmic threshold for this case.
It also shows the closeness of the Tensor PCA and Tensor decomposition problems by handling the recovery of many orthogonal spikes at the same threshold of a single spike: n k/ 4 .
3. T HE MELON GRAPH SIMILAR TO T ENSOR UNFOLDING
Let’s consider the invariant T i 1 ...
i n T i 1 ...i n (illustrated by the graph in Figure 1b when k = 3 ).
Its recovery algorithm (with the matrix obtained by cutting any of the edges) is similar to the tensor unfolding method presented in Richard & Montanari (2014).
The difference is that the melonic algorithm uses only a matrix n × n instead of a matrix n k/ 2 × n k/ 2 for the tensor unfolding.
However, the main contribution of this framework for this graph is that it allows the detection in a linear time ( n k operations for a input (tensor) of size n 3 ) in a constant memory space (it just calculates a scalar).
This provides it a potential usefulness as a ﬁrst step for detecting the signal before deciding to use more computationally costly methods to recover it.
Also, to the best of our knowledge, this framework is the ﬁrst to theoretically prove a conjuncture that the unfolding algorithm works also for the symmetric case.
Theorem 5.
The algorithms 1 and 2 work for the melon graph with β det = β rec =
O ( n k/ 4 ) in linear time and respectively O ( n 2 ) and O (1) memory requirement.
3. T HE TADPOLE GRAPH
Figure 1d has a special characteristic: we can obtain two disconnected parts by cutting only one line.
Therefore, the matrix obtained by cutting that edge is of rank one (in the form of vv T ).
Thus, the vector v has a weak correlation with the signal v 0 , which allow the tensor power iteration ( v i ← T ijk v j v k ) to empirically recover it (formal proofs require to consider some more sophis- ticated variants of power itration like in Anandkumar et al.
(2017) and Biroli et al.
This algorithm is a variant of the already existent homotopy algorithm.
Theorem 6.
The tadpole graph allows to recover the signal vector for k ≥ 3 and β = O ( n k/ 4 ) by using local algorithms to enhance the signal contribution of the vector T ijj .
3. T ENSOR INVARIANT BASED ON THE PERFECT ONE - FACTORIZATION GRAPHS
The next graph we investigate is of degree strictly superior to two.
The perfect one-factorization graph (see Appendix C) has nice combinatorial properties which make it relevant as an example of non trivial graph that we investigate in order to evaluate the richness of the framework. Moreover, these properties also simplify the proofs for its detection and recovery theorems.
Thus, the standard methods involved in the demonstrations of these theorems are instructive for the study of more general graphs.
For k = 3 , it is the tetrahedral graph, which shows very interesting empirical results.
Theorem 7.
The algorithm associated to a perfect one-factorization graph is able to recover the signal vector for β
=
O ( n k/ 4 ) .
3. H ANDLING A SPIKE WITH DIFFERENT DIMENSIONS
We consider the more general case where the tensor T has axes of different dimensions n i ( T ∈ (cid:78) ki =1 R n i ).
We can assume without any loss of generality that n 1 ≥ n 2 ≥ · · · ≥ n k .
[Equation]
where v
[Equation]
Our framework naturally handles this case and allow us to derive a new algorithmic threshold.
It is, to the best of our knowledge, the ﬁrst generalization of the threshold β = n k/ 4 derived in Richard & Montanari (2014) when n i = n ∀ i ∈
[ k ] . Theorem 8.
Using the melon graph, the threshold for v 1 is given by max (cid:16) ( (cid:81)
ki =1 n i ) 1 / 4 , n 1 / 2 1 (cid:17) while the thresholds for v j , j ≥ 2 are equal to ( (cid:81)
ki =1 n i ) 1 / 4 .
This result coincides with the previously known threshold when n i = n . Also the threshold max (cid:16) ( (cid:81) ki
=1 n i ) 1 / 4 , n 1 / 2 1 (cid:17) for v 1 is not surprising.
Indeed, if n 2 = · · · = n k = 1 , the tensor can be seen as a vector of dimension n 1 .
And since the expectation of the norm of a random gaussian vector is n 1 / 2we would need β > n 1 / 2and not β > n 1 / 4to be able to detect the signal vector.
3. H ANDLING M ULTIPLE ORTHOGONAL SPIKES
We consider a symmetric tensor with multiple orthogonal spikes.
[Equation]
where
[Equation]
Theorem 9.
If we have a number of spikes p that is constant in respect to n , we can recover the p spikes by an alternating the use of melonic diagram, power iteration and deﬂation.
We ﬁrst use the melonic diagram to obtain a vector correlated with the signal vectors, then we follow by power iteration to obtain a normalized vector v highly correlated to one of the spikes and then deﬂation which consists in replacing the tensor T by T − α v ⊗ 3 where α = T .
v ⊗ 3 .
The experimental results suggest that it works not only for small values of p but also for a number of spikes up to n .
N UMERICAL EXPERIMENTS
In this section we will investigate the empirical results of the previously mentioned applications in order to see if they match with our theoretical results.
We restrict to the dimension k = 3 for simplicity.
More details about the experiments settings could be found in the appendix.
4. D ETECTION AND COMPARISON OF RECOVERY METHODS
Our detection algorithm, based on the melon graph, uses the fact that the distributions of T ijk T ijk for T =
Z and for T = Z + β v ⊗ 3 become distinct when β is large enough, where the components of Z follows a Gaussian random distribution and v an unitary vector.
This distinction is easily visible, for n = 100 and β = 100 in Figure 4a where we plotted the histograms of the melonic invariant (in blue without signal and in orange with signal) for 500 independent instances of Gaussian random tensors Z .
Thus, to measure the accuracy of the detection of the signal, we use the quantity:- cardinal of the Intersection over the cardinal of the Union (1-IoU).
Fig 4b suggests that a high probability detection requires β ≥ 3 n 3 / 4 .
[Figure]
For the recovery algorithm, we focus in the symmetric case (the most studied case and the most consistent with a symmetric spike) and, as in Richard & Montanari (2014), for every algorithm we use two variants: the simple algorithm and an algorithm where we use 100 power iteration: v i ← T ijk v j v k , which will be distinguished by adding a preﬁx ”p-”.
In Figure 5, we runexperiments for each value of β and plot the 95% conﬁdence interval of the correlation of the vector recovered with the signal vector.
We will compare our method (tetrahedral) to two type of results:
• Other algorithmic methods: the melonic (tensor unfolding) and the homotopy.
To the best of our knowledge, they give the state of art respectively for the symmetric and asymmetric tensor (Biroli et al.
Other methods exist but are either too computationally expensive (sum of squares) or are variants of these algorithms.
• Information-theoretical results: In (Richard & Montanari (2014)), it was proven that computing the global minimum v of the function v (cid:55)→ T ijk v i v j v k recovers the signal vector v 0 above a theoretical threshold β th = 2 .
87 √ n but with exponential time, and that no other approach can do signiﬁcantly better than that.
Thus, to assess the performance of the tetrahedral, we plot in red dashed line denoted ”perf” the deep minimum that is closest to v 0 , by using gradient method with an initialization in v 0 .
Interestingly, we note that not only the tetrahedral is able perform better than other methods, but is able as well to recover the spike at nearly the theoretical threshold even at n ∼ 100 .
4. A SPIKE WITH DIFFERENT DIMENSIONS AND MULTIPLE SPIKES
We aim to recover the three vectors v 1 , v 2[-⊗ v 
-] and v 3 of the spike v 1 ⊗ v 2 ⊗ v 3 .
We repeat 100 times each instance consisting in choosing randomly v 1 , v 2 and v 3 and the Gaussian random tensor Z and we plot the correlation of the signal vectors with the vectors recovered from Z + β v 1 ⊗ v 2 ⊗ v 3 using the tetrahedral.
We see in the Figure 6a that the threshold ( n 1 n 2 n 3 ) 1 / 4 for the three vectors matches perfectly with the experiences when n 3 < n 1 .n 2 .
We also see that when n 3 > n 1 .n 2 the recovery of v 3 (in green and with the diamond and square markers) have a different asymptotic behavior than v 1 and v 2 (it becomes n 1 / 2 3 ), corresponding to what our theoretical study predicted.
In 6b, we take n = 100 and we try to recover 100 orthogonal spikes with β i all equal. The experiment suggests
[Figure]
that we are able to have a good correlation for all the spikes starting from a similar threshold than with only one
[Figure]
C ONCLUSION
In this paper we introduced a novel framework for the tensor PCA based on trace invariants.
Within this framework, we provide different algorithms to detect a signal vector or recover it.
These algorithms use tensor contractions that has a high potential of parallelization and computing optimization.
We illustrate the practical pertinence of our framework by presenting some examples of algorithms and prove their ability to detect and recover a signal vector linearly time memory requirement which could be advantageous in some applications.
Our experimental results show that the tetrahedral graph performs better than the the state of the art in the symmetric case.
Moreover, we also show that two well known algorithms (Homotopy and Tensor Unfolding) can be mapped to our framework and result to simpler graph (e.g. the melonic graph).
Interestingly, our framework is also able to extend the theoretical and practical study of tensor PCA to new and less restrictive situations like data where the dimensions of the axes are different or a sum of orthogonal spikes.
Important directions of future research is to explore the potential of more general graphs and to try to apply these new methods to real data.
A U SEFUL T HEOREMS
The following theorems are needed to the demonstration of the proposed theorems.
• Wedin perturbation : Let M = β v 0 v (cid:62)
+ N a square matrix with (cid:107) v 0 (cid:107) = 1 .
Let v denote the eigenvector with the largest eigenvalue of M .
If β > 2 (cid:107) N (cid:107) op then: 1 −
[Equation]
v , v 0 (cid:105)
[Equation] [Equation]
op β
[Equation]
• Chebyshev inequality : Let X be a random variable with ﬁnite expected value µ and ﬁnite non-zero variance σ 2 .
Then for any real number C > 0 ,
[Equation]
• Theorem (Gelfand’s Formula; 1941):
For any matrix norm (cid:107) .
(cid:107) , we have:
[Equation]
In
particular
we can
choose
the Frobenius norm:
[Equation]
Operator norm of a matrix:
Let A ∈ C n × n be a matrix and λ 1 , . . .
, λ n its (real or complex) eigenvalues.
The spectral radius of
A is its highest eigenvalue in absolute value.
In particular, when a matrix N is symmetric, we have the result ρ ( N ) =
(cid:107) N (cid:107) op .
Thus, in order to compute the operator norm of a given matrix A , we can determine the spectral radius of the matrix A (cid:62) A and take its square root since (cid:13)(cid:13)
A (cid:62) A (cid:13)(cid:13)
op
= (cid:107) A (cid:107) 2 op .
B G AUSSIAN MEASURE
We will follow a similar approach to the one in Gurau (2017) to compute the expectations of trace invariants of a random tensor T whose components are normally distributed.
Let’s consider a graph G of order d associated to a trace invariant
I G ( T ) where T is a tensor of order k .
since the trace invariant is a contraction of pairs of indices of d copies of T , we denote the c -th index of the i -th copy of T by a ic and the set of the d indices of the i -th copy by a j D .
By the deﬁnition of the Gaussian measure we have the equation:
[Equation]
where S ( k ) is the symmetric group (the group of all permutations of a set made of n elements, where we deﬁne a permutation of a set by an arrangement of its elements in a linear order.
For example the permutations of { 1 , 2 , 3 } are (1 , 2 , 3) , (1 , 3 , 2) , (2 , 1 , 3) , (2 , 3 , 1) , (3 , 1 , 2) , (3 , 2 , 1) .).
(cid:80) a indicates a sum over all the indices involved in the computation of the trace invariant, illustrated by a halfedge in the graph (we have k × d indices in total: k indices for each one of the d copies of T of the trace invariant expression).
δ G aa indicates that contracted indices (illustrated by being two ends of the same edge) of the tensors have to be equal.
The term δ a j D a j D τ ( j ) indicates that we set equal the c -th indices of the copy j and τ ( j ) of T for c ∈ { 1 , . . .
, d }
In order to have a clearer representation of the expression in equation 9, in particular the sum over the symmetric group, we introduce the concept of covering graph used in Gurau (2014): a covering graph of G consists in adding d/ 2 new edges of color
(also called propagators) relying pairwise the vertices of G (Figure 7 for an example).
We use the covering graphs to illustrate the permutations τ ∈ S ( k ) .
Then, an edge of color 0 between two vertices i and j will identify all the indices of the i -th and j -th copies of T .
Thus, it will be equivalent to the term δ a j D a j D τ ( j ) .
Using the covering graph, we will be able to rewrite equation 9.
For example, in the case where the edges could only have one color, we have the formula:
[Equation]
Where we denote E c ( G ) the set of edges of color c of G .
Thus {G (cid:48) , G (cid:48) \E 0 ( G (cid:48) )
= G} denotes the graphs which restrict to the graph G when we remove their edges of color 0 , which are by the deﬁnition the covering graphs.
Let c 1 , c 2 ∈ { 0 , . . .
, d } be two different colors of edges.
We denote F c 1 ,c 2 ( G ) the number of closed cycles (that we also call face) of 2 colors of G .
More explicitly, it consists of the number of connected subgraphs left when we keep in G only the edges of colors c 1 , c 2 (see Fig 8 for an illustration).
We denote F ( G ) ≡ (cid:80) c
1 (cid:54) =
c 2 ∈{ 1 ,...,d } .
Then, with a little work (see Gurau (2014) for a proof), we obtain :
[Equation]
This will be the expression we will use to calculate the expectations of our graphs if each edges has only one color.
Example: Let’s calculate the expectation of the tetrahedral graph B :
For each of the three covering graphs, there is one color having two faces (the red in Figure 8) and two colors having only one face.
Hence the expectation of the tetrahedral graph is given by 3 n 4 :
[Equation]
B. T HE CASE OF EDGES OF MIXED COLOR
For a general trace invariant, an edge can have two half edges of different colors.
The propagators (edges of color 0 ) still identify all the indices of their two end vertices.
So we adapt the steps in
Gurau (2014) to obtain equation 12 to this new situation.
More precisely, the cycles that we will count in a covering graph G (cid:48) of G are built by beginning from a half-edge of G then alternating until they go back to the starting half-edge by:
• go from a half-edge of G of color c 1 (cid:54) = 0 to its associated half edge of color c 2 .
• go through the propagator to the half-edge of color c 2 connected to the other end of the propagator.
[Figure]
B. T HE CASE OF A GRAPH WITH AN OPEN EDGE
We encounter this case when we want to study the statistics of a matrix built out of a trace invariant graph.
Indeed cutting an edge of color c removes the contraction of two indices and leaves them free as variables i
and i 2 .
Then in the formula (cid:16)(cid:81) 3 c =1 (cid:81) e c =( v j ,v k ) ∈E c ( G ) δ a cj ,a ck (cid:17) we replace δ a cj ,a ck by δ a cj ,i 1 δ i 2 ,a ck with i
and i 2 ﬁxed.
Thus if a face of color 0 , c contains only one of the two vertices of the edge (if the graph is connected, this only happens when we calculate the variance), its contribution is equal to one.
If it contains the two vertices, it is equal to one if i
= i 2 and zero otherwise.
B. T HE CASE OF A GRAPH WITH A MIX OF SPIKES AND RANDOM TENSORS
We encounter this case when we want to calculate the contribution of the signal to an invariant, which lead us to replace a vertex by the tensorial product of our signal vector.
Since we are interested by an expectation, and knowing that the gaussian distribution is invariant by rotation, we may as well take the signal vector as the unit vector (1 , 0 , . . . , 0) .
Then in the formula (cid:16)(cid:81)
[Equation]
a ck (cid:17) we replace δ a cj ,a ck by δ a cj ,
and we restrict the propagators to the tensors vertices.
First, we notice that if there is an odd number of spikes, the expectation will be equal to zero.
Furthermore, there is no free sum if the index identiﬁed in a cycle ends up with a contraction with a spike.
Its contribution is equal to one.
C T HE PERFECT ONE - FACTORIZATION GRAPH
In the mathematical ﬁeld of graph theory, a graph G is said to be one-factorizable if:
• It is k -regular, which means that each vertex is incident to exactly k edges.
• Its edges can be colored in k colors such that no two incident edges have the same color.
A perfect one-factorization of G happens if, for every two colors c 1 and c 2 of the k colors, restricting
G to the edges of color c 1 and c 2 leads to a connected subgraph cycle (Figure 25).
It is conjectured (Wagner (1992)) that they exist for all n but, until now, they were proven only for prime numbers and up to 56 .
We use this example because of its combinatorial simplicity.
The perfect one-factorization graphs have already been studied in the context of random tensors in Ferrari et al.
[Figure]
D D ETAILS ON THE EXPERIMENTS D. D ETECTION D.1. T HE SETTINGS
For each of the detection ﬁgures, we repeat 500 independent instances of the following experiment:
• We generate randomly the n components of the signal vector v 0 and then normalize v 0 .
• We generate randomly the n 3 components of the random tensor Z .
If we are in the symmetric case, we symmetrize it with the same normalization than Richard & Montanari (2014)):
[Equation]
• We compute the melonic trace invariant Z ijk Z ijk using the numpy method tensordot in Python.
• We compute the tensor T
=
Z + β v ⊗ k •
We compute the melonic trace invariant T ijk T ijk using the numpy method tensordot in Python.
Then in Figure 4a, we plot in blue the distribution of Z ijk Z ijk and in orange the distribution of
T ijk T ijk where T =
Z + β v ⊗ k .
This provides an empirical proof that the distribution of T ijk T ijk is far enough from the one of Z ijk Z ijk to distinguish a tensor with a signal from a Gaussian random tensor.
In Figure 4b, we plot the quantity 1 − cardinal of intersection of the two distributions over the cardinal of the union of the two distributions for different n .
We see that the detection, which is related to the ability to distinguish the two distributions as illustrated in Figure 4a, agrees with the threshold found theoretically O ( n 3 / 4) .
D. R ECOVERY D.2. T HE SETTINGS
For each of the recovery ﬁgures, we repeated 200 times the experiment to draw the error bar (corresponding to a 95% interval conﬁdence).
Then we plot the correlation between the vector returned by the algorithm v and the signal vector v
Each experiment is an independent instance consisting in the following steps:
• We generate randomly the n components of the signal vector v 0 and then normalize it.
• We generate randomly the n 3 components of the random tensor Z .
If we are in the symmetric case, we symmetrize it with the same normalization than Richard & Montanari (2014)):
[Equation]
• We compute the tensor T
=
Z + β v ⊗ k •
We compute our matrix associated to the invariant M constructed from T using the numpy method tensordot in Python.
• We compute its leading eigenvector v • We plot the correlation between the vector resulted from the algorithm v and the signal vector we aim to recover v 0 (namely scalar product < v , v 0 > ).
D.2. A SYMMETRIC TENSOR WITH A SYMMETRIC SPIKE
We see that in the asymmetric case that the homotopy method, represented on our framework by the tadpole graph, performs the best.
It is interesting to note that different graphs have different performances depending on the symmetry of the data and the symmetry of the signal.
[Figure]
T HE NUMBER OF ITERATIONS OF POWER ITERATION FOR EACH METHOD
We plot the number of iterations needed for the power iteration method to converge with an initialization given by the result of one of the methods we compare: tetrahedral, unfolding, homotopy.
[Figure]
Empirical results shows the fast convergence using the power iteration method following the tetrahedral method.
D. M EMORY AND TIME REQUIREMENTS OF THE METHODS General requirements:
Since our method consists on computing algebraic operations on tensors entries to store in matrix n × n , the naive recovery method for a tensor of order k and for a tensor invariant of degree d would require O ( n 2 ) space requirements and O ( n kd ) time complexity.
However, similarly to the matrix case where naive implementation requires n 3 but different methods propose better asymptotic bounds on the time required to multiply matrices, it is possible to simplify the computations by using decomposition on blocks Ma et al.
We hope that there is still a lot of optimization to be done in tensor contractions in the same way it was done for matrix calculations, especially with the rise of the use of tensorial objects.
The ability to be parallelized (each component of the matrix M G ,e can be computed independently) is also an important feature of our methods for practical applications.
Tetrahedral method with ˜ O ( n 4 ) time and ˜ O ( n 3 ) space requirement:
We can calculate the leading eigenvector of the matrix associated to the tetrahedral: M ti 1 i 2 ≡
T
i 1 jk T i 2 j (cid:48) k (cid:48) T ij (cid:48)
k T ijk (cid:48) .
Using the decomposition, we can prove that there is an eigengap between the highest eigenvalue ∼ β 4 and the second one.
Thus, we can use matrix power method with log( n ) steps in order to recover the leading eigenvector v ← M t v .
We can compute it in a time of ˜ O ( n 4 ) and space ˜ O
( n 3 ) .
Indeed this method requires two intermediate matrices we denote A 1 , A 3 and a intermediate 3 − dimensional tensor we denote A 2 , in every step there is at most n 4 operations.
we follow these steps:
• We take an initial vector v •
We compute the matrix A 1 deﬁned as A 1 j (cid:48) k (cid:48)
≡ T i 2 j (cid:48)
[Table]
• We compute the tensor
A 2 deﬁned as A 2 k (cid:48)
ik ≡ A 1 j (cid:48) k (cid:48)
T ij (cid:48)
k • We compute the matrix A 3 deﬁned as as A 3 kj ≡ A 2 k (cid:48)
ik T ijk (cid:48)
• Return the vector v (cid:48) deﬁned as v (cid:48)
i 1 ≡ A 3 kj T i 1 jk
Comparison between different methods
We use our results and the results in Anandkumar et al.
(2017) to ﬁll the Table 1:
[Table]
We plot the empirical time requirement of each method using Python in Figure 13.
However, to have a more accurate estimation of the time requirement, one has to take into account time to read the input (a tensor of size n 3 ).
We obtain in this case Figure 13b.
D. D IFFERENT DIMENSIONS
We see in the following picture that when n 3 > n 1 n 2 , the asymptotic behavior of the recovery threshold of v 3 becomes n 1 /
[Figure]

    n           t ( s ) SWHWU WHWU
SPHO
SPHO KRP
[Figure] [Figure]
M ULTIPLE SPIKES
We see that for any number of spikes inferior to n , we are able to recover at the same threshold than with a single spike
[Figure]
E
[Equation]
ROOFS
Useful lemmas and deﬁnitions:
Lemma 10.
The covering graphs involved in the calculation of E (Tr( M − E ( M )) r ) are the same that are involved in the calculation of E (Tr( M r )) minus the covering graphs where one of the elementary subgraphs has only internal propagators.
Lemma 11.
If β det = O ( n 3 / 4 ) , the intermediate matrices and the intermediate graphs are never dominant.
Calculating the operational norm of a matrix The trace of the power r of a matrix Trace ( M r ) can be represented as the graph gluing open edges of 2 r subgraphs G i representing one matrix A where
M =
A (cid:62) A , we denote this graph by G .
In case we remove the expectation of the matrix, it comes back to the fact that no subgraph G i can have only internal propagators.
Elementary subgraph The subgraph corresponding to M like in Figure.
They are the building bricks.
Internal propagator : When the propagators connect vertices in the same elementary subgraph.
Proof Lemma 1 Let M be the matrix obtained by cuting the edge e of a trace invariant graph G .
We know that the equation 9 identify the indices i and j that are ends of a same edge by including a factor δ ij .
Hence, the non diagonal components M ij , i (cid:54) =
j have a null expectation since the cycle involving the edge e will identify i and j with a δ ij = 0 .
By invariance, E ( M ii ) are equal for all i .
Then, by linearity of the expectation, E ( M ii ) =
E (Tr( M )) /n
=
E ( G ) /n .
[Figure]
Proof theorem
Detecting a signal could be done through calculating the value at T of the invariant associated to G and comparing it to the pure noise distribution of this invariant.
Let n ∈ N and G a graph of degree d .
When β → ∞ , the dominant graph will be the one where all the tensors are replaced by the signal spike, with a value of β d .
The subdominant ones will be the graphs where all but one tensors vertices have been replaced by a spike.
Thus it is easy to show that E ( I G ( T ))
→ β d while σ ( I G ( T )) ∼ β ( d − 1) .
Then, using Chebyshev’s inequality demonstrates that the distributions of I G ( T ) , β → ∞ and I G ( T ) , β
= 0 are completely disjoint which allows the detection of a signal.
Thus, ∃ β det such that ∀ β ≥ β det , the detection is possible.
[Figure] [Figure]
With
[Figure] >> E
[Figure]
Proof theorem 3 Let n ∈ N , G a graph of degree d and M the matrix obtained by cutting the edge e .
When β → ∞ for a ﬁxed n , the dominant matrix by operator norm will be the one where all the vertices are replaced by signal spike, and which is proportional to vv T .
Then applying Wedin perturbation theorem proves that ∃ β rec such that ∀ β ≥ β rec , the recovery is possible.
Proof for theorem 4 To show that (cid:13)(cid:13) M ( N ) − E ( M ( N ) )
(cid:13)(cid:13) op ≥ n dk/ 4 , it is enough to see that the variance of any graph of degree d is superior than dk/ 4 .
Take two copies of a graph G and connect corresponding points from each copy to the other.
The number of cycles will be equal to dk/ 2 .
And the variance will be at least equal to n dk/ 4 (See Figure 19).
Since the expectation is equal to the sum on covering graphs, the variance will always be larger than n dk/ 4 .
As for the variance of
Tr (cid:0) ( MM T ) r (cid:1) it will be always superior to 2 rd .
[Equation] cycle
Proof of theorem 5 : We will use the lemma 10.
Let’s prove by induction that E (Tr (cid:0) ( MM T ) r (cid:1) ) ≤ n kr +1 where M is the matrix associated to the melon by opening one of its edge.
For r = 1 it is straightforward by simply enumerating the cycles.
Now, let r > 1 and let’s assume the statement is true for all r (cid:48) <
r .
Let’s consider one of its elementary subgraphs (deﬁned in the beginning of the section).
Using lemma 10, we can divide the edges of the subgraph in two sets:
• k − 1 edges that are in faces of at least two edges.
• One external edge divided in half edges.
If the k − 1 edges are in faces of exactly two edges, itt means that the propagators at both the left and right vertices are connected to the vertices of the same other elementary subgraph.
This will lead to two possibilities:
• The two elementary subgraphs are next to each other.
Then we remove these two elementary subgraphs and connect the two left open edges.
We will have a contribution of n k from the two elementary subgraphs and use the induction hypothesis to show that the remaining graph has a contribution of maximum n k ( r − 1)+1 .
Combining them we have a total contribution of n k ( r − 1)+1+ k = n kr +1 which proves the theorem.
• The two subgraphs are not next to each other: Then the open edges have also a contribution of maximum 1 / 2 .
Cutting this elementary subgraph and using the induction hypothesis proves the theorem.
Thus using Gelfand’s Formula, the norm operator is equal to n 3 / 2 .
Which gives an recovery threshold for the melon of β = n 3 / 4 using Wedin perturbation.
For the symmetric case, since T symijk = (cid:80) τ ∈ S k T τ ( i ) τ ( j ) τ ( k ) where S k is the symmetric group introduced previously.
We just need to expand the melonic matrix T symi 1 jk T sym i 2 jk in function of the asymmetric tensor T .
We obtain a sum of nine matrices (corresponding to graphs with mixed colors).
We can show that the operational norm of each of these matrices is at most equal to n 3 / 2 .
Proof of theorem 6 : Using the matrix decomposition illustrated in 20, we can see that we have a sum of four matrices of rank one.
Using the usual formula to calculate the expectation of the operational norm of the matrices we obtain:
• (cid:13)(cid:13) M
( N ) (cid:13)(cid:13) =
n• (cid:13)(cid:13) M
( D ) (cid:13)(cid:13) =
n • (cid:13)(cid:13) M ( R ) (cid:13)(cid:13)
=
[Equation]
we can see that the ﬁrst one dominates by n 1 / 4 the last one (consisting in the signal vector) so we are able to use power iteration or one of its variants (like in Anandkumar et al.
(2017) and Biroli et al.
to recover it.
[Equation]
Proof theorem 7 : Let’s deﬁne a set of even numbers ( s i ) i ∈{ 0 ,...,r } where s i / 2 is equal to the number of propagators having its two vertices in the subgraph G i (we call these propagators internal).
Thus we have s/ 2 ≡ =
(cid:80) i s i / 2 internal propagators in total.
We can divide the edges of the subgraph G in three classes.
Each edge could be part of a face of either one edge, two edges, three
edges
, etc
Let’s count how many we can have of each type in an optimal conﬁguration (maximizing the number of faces) for each class.
• Edges only connected to internal propagators, there is at most (cid:80)
i s i ( s i − 1) / 2 such edges.
– We can have a maximum of s i / 2 faces of one edge since that is the number of internal edges.
– Then, from the s i ( s i − 2) / 2 remaining ones, if we want to put them in faces of two edges, the two edges needs too to be in the same subgraph.
Thus, we can create a maximum of s i / 2( s i / 2 − 1) /
= s i
( s i − 2) / 8 different couples of propagators.
Each couple can’t create more than one face since otherwise it will contradict
the maximally
single trace hypothesis of the graph.
So there is a maximum of s i ( s i − 2) / 4 edges that can be put in two edges faces.
– The remaining s i ( s i − 2) / 4 will be in faces of at least three edges.
• Edges connected to one internal and one external propagators, there is at most (cid:80)
i s i ( k + 1 − s i ) one such edges.
They can only be in faces of at least three edges.
• Edges only connected to external propagators, there is at most (cid:80)
[Equation]
k + 1 − s i
[Equation]
( k −
[Equation]
2) such edges.
They can only be put in faces of at least two edges.
• The open edge can be put in a face of maximum one edge.
So we have a maximal number of faces given by:
[Equation]
Let’s focus in the case where s i = 0 .
If all the points of the graph are connected to points of another graph, then the two extremes of the open edge connects to the same graph.
In that case the open edge has a contribution of only 1 / 2 .
In the other case, then there is at least k edges which contributes only by at most 1 / 3 making them lose 1 / 6 .
Either way the contribution of the subgraph can’t go further then k ( k + 1) / 4 .
We obtain:
[Equation]
We can easily check that this quantity is always smaller than k ( k +1) for k > 5 .
Proof of theorem 8 : Let’s consider the melonic graph with the edge associated to the index of the space of dimension n l open.
It is associated to the matrix:
[Equation]
For simplicity the graphs will be drawn for k = 3 .
We will treat two separate cases:
• Case n l ≤ (cid:89)
[Equation]
Let’s prove by induction the statement: S
[Equation] [Equation]
– Initialisation with r = 2 : For M − E ( M ) we have the following graph and its covering graphs:
E (Tr( M r ) =
n l k (cid:89) i = n i + k
(cid:89)
i
[Equation]
n i ≤ 2 n l k (cid:89) i = n r/ 2 i
– Assume S r is true for all r (cid:48) <
r , let’s prove it for r .
* In the case where there is no cycle of color l of length 1 (which means a propagator that shares the same ends that an edge of color l ), then all edges have a contribution of 1 / 2 at maximum.
So the contribution of that covering graph will be at most equal to (cid:81) ki
=1 n r/ 2 i .
In case there is such a propagator, which means that there is a propagator coinciding with an edge of color l between two elementary subgraphs.
Let r
Then either the two other ends of the two elementary subgraphs are connected by a propagator like in 24 or not.
In the ﬁrst case, the total number of cycles will be at most equal to one cycle of each color, plus the number of cycles of a graph with 2( r − 1) elementary subgraphs, so E (Tr (cid:0) M 2 r (cid:1) ≤ 2 n l (cid:81)
ki =1 n ri .
, we can remove cycle
[Equation]
Let’s prove by induction the statement: S r : E
[Tr(( M − E ( M )) r )]
= O ( n rl (cid:81) i (cid:54) =
l n i ) .
– Initialisation with r = 2 : For M − E ( M ) we have the covering graphs in 21 which checks S– Assume S r is true for all r (cid:48) <
r , let’s prove it for r .
* In the case where there is no cycle of color l of length 1 (which means a propagator that shares the same ends that an edge of color l ), then all edges have a contribution of 1 / 2 at maximum.
So the contribution of that covering graph will be at most equal to (cid:81) ki
=1 n r/ 2 i .
In case there is such a propagator, which means that there is a propagator coinciding with an edge of color l between two elementary subgraphs.
Let r
Then either the two other ends of the two elementary subgraphs are connected by a propagator like in 24 or not.
In the ﬁrst case, the total number of cycles will be at most equal to one cycle of each color, plus the number of cycles of a graph with 2( r − 1) elementary subgraphs, so E
[Equation]
[Figure]
[Equation]
Proof of theorem 9 : Let ˜ A
= M e G ( T ) the matrix we have with noise and A = (cid:80) pi =1 β di v ki the matrix when there is no noise and only signal.
We call an eigenspace of A the subspace of R n spanned by some eigenvectors of A .
The idea of the proof of the theorem is that for β large enough, the eigenspace spanned by the signal vectors S should be close to the eigenspace spanned by the p largest eigenvectors of ˜ A denoted ˜ S .
Then, we take a vector from this subspace ˜ S , we perform power iteration on it in order to recover a signal vector w 1 , and repeat this step by restricting to the subspace of ˜ S orthogonal to w 1 .
If there were only one signal, there is a simple way to calculate how close are v 1 and w 1 : θ ( v 1 , w 1 ) =
cos − 1 ( v 1 .w 1 ) .
To quantify the closeness of two subspaces S and ˜ S , we use the canonical angles.
The ﬁrst canonical angle is deﬁned as follow:
[Equation]
where we denoted the maximizers by x 1 and y 1 and x.y denotes the scalar product between x and y .
Then we deﬁne by iteration the k th canonical angle by:
[Equation]
We put these angles in a matrix Θ
[Equation]
We deﬁne now the distance between S and ˜ S by (cid:107) sin(Θ)
(cid:107) F with sin(Θ) deﬁned as:
[Equation]
The Davis-Kahan Theorem uses these tools to provide a bound on subpaces spanned by eigenvectors under perturbation (Stewart & guang Sun (1990)):
Davis-Kahan: Let A , A (cid:48) two symmetric matrices with respective eigenvectors:
[Equation]
Let’s deﬁne the eigengap as:
[Equation]
Then if δ >
[Equation]
δ (
[Equation]
The result also holds for (cid:107) .
(cid:107) op
Let’s study the case where β d > cn k/ 4 where c is a constant large enough.
The gap will be equal to δ =
( c − 1) n k/ 4 .
In the case of the melonic graph, (cid:13)(cid:13)(cid:13) ˜
A − A (cid:13)(cid:13)(cid:13) op can be bounded by 2 n k/ 4 .
Let’s take a vector z from ˜ S .
It can be put in the form of:
[Equation]
where q (cid:48) and q are unitary vectors, and q = q (cid:48) − (cid:80) pi
=1 ( q (cid:48) .
x k ) x
k
so it is orthogonal to all the vectors v i and that |
γ 0 | ≤ |
α 0 | .
Let’s put a high bound for alpha 0 .
Since x k .y k
= cos( θ k ) , ∀ k ∈ { 1 , . . .
, p }
[Equation]
Using these results, we show that power iteration on a vector z of S is able to recover a signal vector v k :
[Equation]
Thus, repeating z ← T zz will converge to v l max with l max = max( { β l γ 2 l , l ∈ { 1 , . . .
, r }} )
We repeat this step p times, but before each step we use deﬂation by computing:
[Equation]
Proof Lemma
The covering graphs involved in the calculation of E (Tr( M − E ( M )) r ) are the same that are involved in the calculation of E (Tr( M r )) minus the covering graphs where one of the elementary subgraphs has only internal propagators.
We can ﬁrst begin by showing that if we have two matrices M 1 of degree d 1 and M 2 of degree d 2 , then the covering graphs involved in Tr(( M 1 − E ( M 1 ))
M 2 ) are the covering graphs of
Tr( M 1 M 2 ) minus the ones where the propagators are internal.
Let’s denote the color of the open edge by l .
To distinguish the d 1 + d 2 vertices and their indices, we assign numbers to each vertex.
Namely the vertices of M 1 are denoted by 1 , . . .
, d 1 with 1 and 2 associated to the two vertices adjacent to the open edge of M 1 , and d 1 , . . .
, d 1 + d 2 where the vertices of the open edge of Mare d 1 + 1 , d 1 + 2 .
We denote all the indices by a , and the indices of all colors associated to a vertex l by a D l .
A subset of the symmetric group σ ( d 1 + d 2 ) are the permutations which could be written as τ 1 τwhere τ 1 is a permutation of (1 , . . .
, d 1 ) and τ 2 is a permutation of ( d 1 , . . .
, d 1 + d 2 ) .
We denote by the symbol δ B aa the δ of all the indices that are contracted in the graph.
Let B 1 the graph associated to M 1 and B 2 the graph associated to M 2 .
Let B (cid:48) the subgraph of B obtained by opening an edge of color c corresponding to the vertices i and j , it is an elementary subgraph of B r , the graph corresponding to Tr( M r ) .
We have
[Equation] [Equation]
Since the edges of B are the union of the edges of B 1 and B 1 and an edge of color l connecting the vertices of the open edges: 1 with d 1 +1 and 2 with d 1 +2 , we have: δ B aa
[Equation]
This let us write:
E (Tr
[Equation]
M
[Equation]
= (cid:88) a δ B aa (cid:20) (cid:88)
τ ∈ S ( d 1 + d 2 ) d 1 + d 2 (cid:89) l = δ a l D a D τ ( l ) − (cid:88) τ 1 ∈ S ( d 1 ) ,τ
2 ∈ S ( d 2 ) d 1 (cid:89) l = δ a l D a D τ 1( l ) d 1 + d 2 (cid:89) m =
d 1 + δ a m D a D τ 2( m ) (cid:21) , (28)
Thus the only permutations that are not cancelled are the permutations that does not live invariant the subset { 1 , . . .
, d 1 } , which means that ∃ i ∈ { 1 , . . .
, d 1 } such that τ ( i ) ∈ { d 1 + 1 , . . .
, d 1 + d 2 } .
These permutations correspond to the covering graphs that does not have only internal propagators connecting the vertices of M 1 .
This proves our claim.
[Figure]
Proof of lemma 11 : Let G a graph that has β dec = n k/ 4 as a detection threshold.
Let’s assume that an intermediate graph G in with r spikes dominate for β > β dec and let’s take the covering graph of G in 2 which gives the contribution for the domination.
We cut the the propagators that involves the spikes and connect them together, while we connect by a new propagator each vertex of spike to its mirror.
This covering graph will give a strictly superior contribution to the expectation if the graph is connected since:
• The cycles that didn’t involve a spike stay unchanged •
The cycles that involved a spike adds together a new contribution of n α with α > 3 r/ 4 .
Thus for n 3 / 4 << β << n α , the pure noise graph dominates the intermediate graph, which leads to a contradiction.
The demonstration for the matrices M follows the same steps but instead of G 2 , one has to consider Tr( M r ) .