A NEW FRAMEWORK FOR TENSOR PCA BASED ON TRACE INVARIANTS
Anonymous authors
Paper under double-blind review
A BSTRACT
We consider the Principal Component Analysis (PCA) problem for tensors T ∈
( R n ) ⊗ k of large dimension n and of arbitrary order k ≥ 3 .
It consists in recovering a spike v ⊗ k(related to a signal vector v 0 ∈ R n ) corrupted by a Gaussian noise tensor Z ∈ ( R n ) ⊗ k such that T
= β v ⊗ k
+ Z where β is the signal-to-noise ratio.
In this paper, we propose a new framework based on tools developed by the theoretical physics community to address this important problem.
They consist in trace invariants of tensors built by judicious contractions (extension of matrix product) of the indices of the tensor T .
Inspired by these tools, we introduce a new process that builds for each invariant a matrix whose top eigenvector is correlated to the signal for β sufﬁciently large.
Then, we give examples of classes of invariants for which we demonstrate that this correlation happens above the best algorithmic threshold ( β ≥ n k/ 4 ) known so far.
This method has many algorithmic advantages: (i) it provides a detection algorithm linear in time and has only O (1) memory requirements (ii) the algorithms are very suitable for parallel architectures and have a lot of potential of optimization given the simplicity of the mathematical tools involved (iii) experimental results show an improvement of the state of the art for the symmetric tensor PCA.
Furthermore, this framework allows more general applications by being able to theoretically study the recovery of a spike in the form of v 0 ⊗ · · · ⊗ v k with different dimensions ( T ∈ R n 1 × n 2 ×···× n k with n 1 , . . . , n k ∈ N ) as well as the recovery of a sum of different orthogonal spikes.
We provide experimental results to these different cases that match well with our theoretical ﬁndings.
I NTRODUCTION
Powerful computers and acquisition devices have made it possible to capture and store real-world multidimensional data.
For practical applications (Kolda & Bader (2009)), analyzing and organizing these high dimensional arrays (formally called tensors) lead to the well known curse of dimensionality.
Thus, dimensionality reduction is frequently employed to transform a high-dimensional data set by projecting it into a lower dimensional space while retaining most of the information and underlying structure.
One of these techniques is Principal Component Analysis.
PCA has made remarkable progress in a large number of areas thanks to its simplicity and adaptability (Jolliffe &
Cadima (2016)).
In the Tensor PCA, as introduced by Richard & Montanari (2014), we consider a simple model where we attempt to detect and retrieve an unknown unit vector v 0 from noise-corrupted multilinear measurements put in the form of a tensor T .
Using the notations found below, our model consist in:
[Equation]
with Z a pure Gaussian noise tensor of order k and dimension n with identically independent distributed (iid) standard Gaussian entries:
Z i 1 ,i 2 ,...,i k ∼ N (0 , 1) and β the signal-to-noise ratio.
To solve this important problem, many methods have been proposed.
However, practical applications require optimizable and parallelizable algorithms that are able to avoid the high computationally cost due to an unsatisfactory scalability of some of these methods.
A summary of the time and space requirement of some existent methods can be found in Anandkumar et al.
One way to achieve this parallelizable algorithm is through methods based on tensor contractions (Kim et al.
(2018)) which are extensions of the matrix product.
These last years, tools based on tensor contractions have been developed by theoretical physicists where random tensors have emerged as a generalization of random matrices.
We want to investigate the algorithmic threshold of tensor
PCA and some of its variants using the theoretical physics approach in the hope that it leads to new insights and knowledge in tensor PCA.
Tensor PCA and tensor decomposition (the recovery of many spikes addressed in Section 3.5) is motivated by the increasing number of problems in which it is crucial to exploit the tensorial structure Sidiropoulos et al.
Recently it was successfully used to address important problems in unsupervised learning (learning latent variable models, in particular latent Dirichlet allocation
Anandkumar et al.
(2014), Anandkumar et al.
(2015)), supervised learning (training of two-layer neural networks, Janzamin et al.
and reinforcement learning (Azizzadenesheli et al.
Moreover, we note that some of our results tends to generalize the applications of the methods to more practical settings like the case of a tensor with axes of different dimensions (adequate for data which are inherently asymmetric like a video).
Related work Tensor PCA was introduced by Richard & Montanari (2014) where the authors suggested and analyzed different methods to recover the signal vector like matrix unfolding and power iteration.
Since then various other methods were proposed.
Hopkins et al.
(2015) introduced algorithms based on the sum of squares hierarchy with the ﬁrst proven algorithmic threshold of n k/ 4 .
However they generally require high computing resources and complex mathematics.
Other studied methods have been inspired by different perspectives like homotopy in Anandkumar et al.
(2017), statistical physics (Arous et al.
(2020), Ros et al.
(2019), Wein et al.
and Biroli et al.
quantum computing (Hastings (2020)) as well as statistical query (Dudeja & Hsu (2020)).
Recently, a fundamentally different set of mathematical tools that have been developed for tensors in the context of high energy physics have been used to approach the problem.
They consist in trace invariants of degree d, d ∈ N , obtained by contracting pair of indices of d copies of the tensor T .
They have been used in Evnin (2020) to study the highest eigenvalue of a real symmetric Gaussian tensor.
Subsequently, Gurau (2020) provided a theoretical study on a function based on an inﬁnite sum of these invariants.
Their results suggest a transition phase for the highest eigenvalue of a tensor for β around n 1 / 2 in a similar way to the BBP transition in the matrix case (Baik et al.
Thus, this function allows the detection of a spike.
However evaluating the function involves an computing an integral over a n-dimensional space, which may not be possible in a polynomial time.
The contribution of this paper is the use of these invariant tools to build tractable algorithms with polynomial complexity.
In contrast to Gurau (2020), instead of using a sum of an inﬁnite number of invariants, we select one trace invariant with convenient properties to build our algorithms.
It lets us detect the presence of the signal linearly in time and with a space requirement in O (1) .
Moreover, in order to recover the signal vector besides simply detecting it, we introduce new tools in the form of matrices associated to this speciﬁc invariant.
Within this framework, we show as particular cases, that the two simpler graphs (of degree two) are equivalent to the tensor unfolding and to homotopy algorithm (which is equivalent to average gradient descent).
These two algorithms are the main practical ones known from the point of view of space and time requirement (Anandkumar et al.
(2017) provides a table comparison).
Interestingly, our framework allows us to go further by using graphs of higher degrees and we can see in that case that the tetrahedral graph (degree four) performs better in the symmetric case.
We should notice that the complexity of the algorithm does not necessarily increase with the degree of the graph (unlike sum of squares hierarchy).
Notations We use bold characters T , M , v for tensors, matrices and vectors and T ijk , M ij , v i for their components.
[ p ] denotes the set { 1 , . .
. , n } .
A real k − th order tensor T ∈ (cid:78) ki =1 R n i is a member of the tensor product of Euclidean spaces R n i , i ∈
[ k ] .
It is symmetric if T i 1 ...
i k
=
T τ ( i 1 ) ...
τ ( i k ) ∀ τ ∈ S k where S k is the symmetric group (more details are provided in Appendix B).
For a vector v ∈ R n , we use v ⊗ p ≡ v ⊗ v ⊗ · · · ⊗ v ∈ (cid:78) p R n to denote its p -th tensor power.
(cid:104) v , w (cid:105) denotes the scalar product of v and w .
Let’s deﬁne the operator norm, which is equivalent to the highest eigenvalue of a tensor of any rank: (cid:107) X (cid:107) op ≡ max { X i 1 ,...,i k ( w 1 ) i 1 . . .
( w k ) i k , ∀ i ∈ { 1 , . . . , n } , (cid:107) w i (cid:107) ≤ 1 } Let A ∈ C n × n be a matrix and λ 1 , . . .
, λ n its (real or complex) eigenvalues.
The spectral radius of A is its highest eigenvalue in absolute value.
In particular, when a matrix N is symmetric, we have the result ρ ( N ) =
(cid:107) N (cid:107) op .
Thus, in order to compute the operator norm of a given matrix A , we can determine the spectral radius of the matrix A (cid:62) A and take its square root since (cid:13)(cid:13)
A (cid:62) A (cid:13)(cid:13)
op
= (cid:107) A (cid:107) 2 op .
The trace of A is denoted Tr( A ) .
We denote the expectation of a variable X by E ( X ) and its variance by σ ( X ) .
We say that a function f is negligible compared to a positive function g
and we write f = o ( g ) if lim n →∞ f/g → 0 .
We say that algorithm detects or recovers a signal from a tensor, when it is able to do so with a probability p ( n ) → 1 in the large n limit.
Einstein summation convention It is important to keep in mind throughout reading the paper that we will follow the Einstein summation convention: when an index variable appears twice in a single term and is not otherwise deﬁned, it implies summation of that term over all the values of the index:
T ijk T ijk ≡ (cid:80) ijk T ijk T ijk .
It is a common convention when addressing tensor problems that helps make the equations more comprehensible
G ENERAL F RAMEWORK FOR S IGNAL D ETECTION AND R ECOVERY 2. W HAT DO WE USE TO STUDY THE SIGNAL ?
An important concept in problems involving matrices is the spectral theory.
It refers to the study of eigenvalues and eigenvectors of a matrix.
It is of fundamental importance in many areas.
In machine learning, the matrix PCA computes the eigenvectors and eigenvalues of the covariance matrix of the features to perform a dimensional reduction while ensuring most of the key information is maintained.
In this case, the eigenvalues is a very efﬁcient tool to describe data variability.
In the case of signal processing, eigenvalue can contain information about the intensity of the signal, while the eigenvector points out to its direction.
Lastly, a more theoretical example involves quantum physics where the spectrum of the matrix operator is used to calculate the energy levels and the state associated.
In all of these examples, an important property of the eigenvalues of a n-dimensional matrix M is its invariance under orthogonal transformations { M → OMO − 1 , O ∈ O( n ) } where O( n ) is the n -dimensional orthogonal group (i.e. the group of real matrices that satisﬁes OO (cid:62)
=
I n ).
Since these transformations essentially just rotate the basis to deﬁne the coordinate system, they must not affect intrinsic information like data variability, signal intensity or the energy of a system.
The eigenvalues are able to capture some of these inherent information, but recovering the complete general information requires computing their respective eigenvectors (for example to ﬁnd the principal component, the direction of the signal or the physical state).
There is more such invariants than eigenvalues.
Another important set worth mentioning are the traces of the n ﬁrst matrix powers
Tr( A ) , Tr (cid:0)
A 2 (cid:1) , . . . ,
Tr( A n ) .
Obtaining them uses slightly different methods than eigenvalues, but they contain the same information since each set can be inferred from the other through some basic algebraic operations.
On the basis of the matrix case, we expect that for a tensor T ∈ (cid:78)
ki =1 R n i , tensor quantities that are invariant under orthogonal transformations ( T a 1 j ...
a
[Equation]
→
[Equation] [Equation]
[Equation] [Equation]
1 j ...b kj for O ( i ) ∈
O( n i ) ∀ i ∈
[ k ] . ) should capture similar intrinsic information like the intensity of the signal, and conceivably, there should be other objects related to these quantities that are able to indicate the direction of the signal.
However, the concept of eigenvalue and eigenvector is ill deﬁned in the tensor case and not practical giving that the number of eigenvalues is exponential with the dimension n (Qi (2005), Cartwright & Sturmfels (2013)) and computing them is very complicated.
In contrast, we have a very convenient generalization of the traces of the power matrices for the tensors that we call trace invariants.
They have been extensively studied during the last years in the context of high energy physics and many important properties have been proven.
We will ﬁrst give a more formal deﬁnition of trace invariants.
Let T be a tensor whose entries are
T i 1 ,...,i k .
Let’s deﬁne a contraction of a pair of indices as setting them equal to each other and summing over them, as in calculating the trace of a matrix ( A ij → (cid:80) ni
=1 A ii ).
The trace invariants of the tensor T correspond to the different way to contract pairs of indices in a product of an even number of copies of T .
The degree of the trace invariants consists in the number of copies of T contracted.
For example, (cid:80) i 1 ,i 2 ,i 3 T
i 1 i 2 i 3 T i 1 i 2 i 3 and (cid:80) i 1 ,i 2 ,i 3 T i 1 i 2 i 2 T i 1 i 3 i 3 are trace invariants of degree 2.
In the rest of this paper, we will use the Einstein summation convention by dropping the sum symbol and imply summation when an index variable appears twice.
A trace invariant of degree d of a tensor T of order k admits a practical graphical representation as edge colored graphs.
We draw d vertices representing the d different copies of T .
The indices of each copy is represented by k half-edges with a different color for each index position as shown in Figure 1a.
When two different indices are contracted, we connect their corresponding half-edges.
For each graph G , we denote the invariant associated to it I G .
Three important examples of trace invariants are invariants associated to the melon diagram (Fig. 1b), the tetrahedral (1c) and the tadpole (1d).
Avohou et al.
(2019) provides a thorough study about the number of trace invariants for a given degree d .
A very useful asset of these invariants is that we are able to compute their expectation for tensors whose components are Gaussian using simple combinatory (Gurau (2017)).
[Figure]
2. W HAT DO WE USE TO RECOVER THE SIGNAL ?
As previously mentioned in Section 2.1, an invariant should be able to detect a signal.
But if our goal is to recover it, we should ﬁnd mathematical objects that are able to provide a vector.
To this effect, we introduce in this paper a new set of tools in the form of matrices.
They are obtained by cutting an edge e of the graph in two half edges, which amounts to not summing over the two indices i
and i 2 associated to these two half-edges and using them to index a matrix instead (see Figure 3 for an example).
We denote this matrix M G ,e .
We will drop the index of the matrix when the context is clear.
[Figure]
[Equation]
[Figure]
2. P HASE T RANSITION WITHIN THIS F RAMEWORK
We can represent the tensor from which we hope to extract the signal represented graphically as:
[Equation]
This decomposition leads us naturally to decompose in a similar way a tensor trace
invariant I G ( T ) into two parts, separating a ﬁrst contribution I ( N ) G ( T ) associated to the pure noise tensor contribution and a second part I ( S ) G ( T ) enclosing all the other contributions, which are resulting from the addition of a signal tensor.
[Equation]
An identical decomposition can be carried out for the matrix.
Let’s consider a graph G and its associated trace invariant I G .
Let’s denote I (cid:48)
G the invariant associated to the subgraph obtained by removing from G the edge e and its two ends.
We can distinguish three kind of contributions to the matrix M G ,e illustrated in Figure 3 (where we denoted the invariant I (cid:48)G by I (cid:48) for simplicity).
In what follows, we use the superscript ( N ) when we want to stress that we are talking about pure random tensors, in order to compute expectations and variances.
[Figure] [Figure]
I
[Figure]
I
[Figure] [Figure]
[Equation]
Lemma 1.
[Equation]
Using the lemma 1, for a general graph, we identify three possible phases in function of which matrix operational norm is much larger than the others: If (cid:13)(cid:13) M ( N ) − E ( M ( N ) ) (cid:13)(cid:13) op (cid:29) (cid:13)(cid:13) M ( D ) (cid:13)(cid:13) op , (cid:13)(cid:13)
M ( R ) (cid:13)(cid:13) op then no recovery, no detection, we can’t distinguish if there is a signal.
It is for example the phase for β → 0 .
If (cid:13)(cid:13) M ( D ) (cid:13)(cid:13)
op (cid:29) (cid:13)(cid:13) M ( N ) − E
M ( N ) )
(cid:13)(cid:13) op , (cid:13)(cid:13)
M ( R ) (cid:13)(cid:13) op then detection but no recovery.
We can detect the presence of the signal (thanks to the highest eigenvalue) but we can’t recover the signal vector since the eigenvector associated to the maximal eigenvalue is not correlated to the signal vector.
Detection and recovery: (cid:13)(cid:13)
M ( R ) (cid:13)(cid:13) op (cid:29) (cid:13)(cid:13)
M ( N ) − E ( M ( N ) )
(cid:13)(cid:13) op , (cid:13)(cid:13) M
( D ) (cid:13)(cid:13) op .
We recover the signal vector.
It is for example the phase for β → ∞ .
2. A LGORITHMIC T HRESHOLD FOR A G ENERAL G RAPH
We can now state the important algorithms that will be essential for this paper.
Algorithm 1: Detection algorithm associated to the graph G and edge e
Input: The tensor T = β v ⊗ k + Z
Goal: Detection of v
1 Calculate the scalar
= I G (
T
2 Compare
Result: Gives a probability of the presence of a spike
Let’s clarify that the claims concern the large n limit and that empirically the approximation of large n limit seems valid for n > 25 .
The ﬁrst algorithm gives a criteria for distinguishing a pure noise tensor from a tensor with a spike.
Denoting the expectation and the variance of the trace invariant associated to a graph G for a gaussian random tensor model by E ( I ( N ) G ) and σ ( I ( N ) G ) .
The algorithm consists simply in calculating the trace invariant of the tensor and comparing its distance from E ( I ( N ) G ) with σ ( I ( N ) G ) .
It is straightforward to see that calculating a trace invariant (which is a scalar) like T ijk T ijk only needs O (1) memory.
Theorem 2.
Let G be a graph of degree d , ∃ β det > 0 so that Algorithm 1 detects the presence of a signal for β ≥ β det .
The second algorithm is able to recover the spike through the construction of the matrix of size n × n
M G ,e associated to a given graph G and edge e .
Input: The tensor T = β v ⊗ k + Z
Goal: Estimate v 1 Calculate the matrix M G ,e ( T ) ; 2 Compute its top eigenvector by matrix power iteration (repeat v i ← M i jv j ).
Result: Obtaining an estimated vector v
Theorem 3.
Let G be a graph of degree d , ∃ β rec > 0 so that Algorithm 2 gives an estimator v so that < v , v 0 > = 1 − o (1) for β ≥ β rec .
The last theorem gives a lower bound to the threshold above which we can detect and recover a spike using a single graph.
Interestingly, this threshold which appears naturally in our framework, matches the threshold below which there is no known algorithm that is able to recover the spike in polynomial time.
Theorem 4.
Let k ≥ 3 .
It is impossible to detect or recover the signal using a single graph below the threshold β ≤ n k/ 4 which is the minimal variance of any graph.
S OME APPLICATIONS OF THIS FRAMEWORK
Using these algorithms, we are now able to investigate the performance of our framework in various theoretical settings.
In the ﬁrst two subsections, we study the algorithms associated to two trace invariants of degree 2.
They consist of the melonic diagram, which gives a very practical detection algorithm, and whose recovery algorithm is a variant of the unfolding algorithm, and the tadpole diagram which gives a similar algorithm to the homotopy algorithm.
The third subsection goes further in terms of graph’s degree and investigate the algorithms associated to the perfect one-factorization graphs (consisting in the tetrahedral when k = 3 ).
The last two sections are an illustration of the versatility of this framework.
We study the case n 1 (cid:54) =
n 2 (cid:54) = n 3 which is important for practical applications where the dimensions are naturally asymmetric.
Our methods allows us to derive a new algorithmic threshold.
It also brings closer the Tensor PCA and Tensor decomposition problems by handling the recovery of many orthogonal spikes at the same threshold of a single spike: n 3 / 4 .
3. T HE MELON GRAPH SIMILAR TO T ENSOR UNFOLDING
Let’s consider the invariant T i 1 ...
i n T i 1 ...i n (illustrated by the graph in Fig.
1b when k = 3 ).
Its recovery algorithm (with the matrix obtained by cutting any of the edges) is similar to the tensor unfolding method presented in Richard & Montanari (2014).
The difference is that the melonic algorithm uses only a matrix n × n instead of a matrix n k/ 2 × n k/ 2 for the tensor unfolding.
However, the main contribution of this framework for this graph is that it allows the detection in a linear time (there is three sums to perform so n 3 operations, and the tensor input is of size n 3 ) in a constant memory space (it just calculates a scalar).
Giving it a potential interest as a ﬁrst step for detecting the signal before deciding to use more computationally costly methods.
Also, to the best of our knowledge, this framework is the ﬁrst to theoretically prove a conjuncture that the unfolding algorithm works also for the symmetric case.
Theorem 5.
The algorithms 1 and 2 works for the melon with β det =
=
O ( n k/ 4 ) , β th = O ( n k/ 4 ) in linear time and respectively O ( n 2 ) and O (1) memory requirement.
3. T HE TADPOLE GRAPH
Figure 1d has a special characteristic: we can obtain two disconnected parts by cutting only one line.
The consequence of this characteristic is that the matrix obtained by cutting that edge is of rank one (in the form of vv T ).
Thus, the vector v has a weak correlation with the signal v 0 , which allow variants of tensor power iteration ( v i ← T ijk v j v k ) to recover it (simple power iteration is also able to recover the signal vector, but a formal proof needed to consider some more sophisticated variants of it like in Anandkumar et al.
(2017) and Biroli et al.
It is a variant of the already existent homotopy algorithm.
Theorem 6.
The tadpole graph allows to recover the signal vector for k ≥ 3 and β = O ( n k/ 4 ) by using local algorithms to enhance the signal contribution of the vector T ijj (corresponding to the half graph resulting from cutting the middle edge).
3. T ENSOR INVARIANT BASED ON THE PERFECT ONE - FACTORIZATION GRAPHS
The next graph we investigate is of degree strictly superior to two.
We investigate this example because it has nice combinatorial properties which simplify the proofs to its detection and recovery theorems.
Also, the demonstrations of these theorems involve standard methods that can be used to more general graphs.
It is a generalization of the tetrahedral graph and it shows very interesting empirical results.
Theorem 7.
The complete graph is able to recover the signal vector for β
=
O ( n k/ 4 ) .
3. H ANDLING A SPIKE WITH DIFFERENT DIMENSIONS
We take k =
and we consider the more general case where the tensor has axes of different dimensions n i where n i ∈ N .
We can assume without any loss of generality that n 1 ≥ n 2 ≥ n 3 .
[Equation]
where
[Equation]
One way to obtain the spike in this case is to insert the tensor in a new symmetric tensor T sym ∈
[Equation]
n 3 (deﬁned as T sym i n 1 + j n 1 + n 2 + k = T ijk for i ∈ [ n 1 ] , j ∈ [ n 2 ] , k ∈ [ n 3 ] and the other components of T sym are obtained by symmetry) and then apply our usual methods to recover a vector from which we can extract the three signal vectors, the threshold to recover the three vectors would be equal to ( n 1 + n 2 + n 3 ) 3 / 4 ).
However our framework naturally handles this case without resorting to symmetrization.
We have the following theorem:
Theorem 8.
Assume n 1 ≥ n 2 ≥ n 3 without any loss of generality.
Using the melon graph, the threshold for v 1 is given by max (cid:16) ( n 1 n 2 n 3 ) 1 / 4 , n 1 / 2 1 (cid:17) while the thresholds for v 2 and v 3 are equal to ( n 1 n 2 n 3 ) 1 / 4 .
To the best of our knowledge, the threshold max (cid:16) ( n 1 n 2 n 3 ) 1 / 4 , n 1 / 2 1 (cid:17) for v 1 and ( n 1 n 2 n 3 ) 1 /for v 2 , v 3 is the new best known threshold for this case.
Not only it coincides with the previous threshold when n 1 = n 2 = n 3 , but also the result for n 1 > n 2 n 3 seems logical.
Indeed, in the limit case of n 2 = n 3 = 1 , the tensor will be a vector.
And since the expectation of the norm of a random gaussian vector is n 1 / 2 we would need β > n 1 / 2 to be able to detect the signal vector.
3. H ANDLING M ULTIPLE ORTHOGONAL SPIKES
We take k =
and we consider a symmetric tensor with multiple orthogonal spikes.
[Equation]
where
[Equation] [Equation]
Theorem 9.
If we have a number of spikes p that does depend on n , we can recover the p spikes by an alternation of the melonic diagram,
We ﬁrst use the melonic diagram to obtain a vector correlated with the signal vectors, then we follow by power iteration to obtain a normalized vector v highly correlated to one of the spikes and then deﬂation which consists in replacing the tensor T by T − α v ⊗ 3 where α = T .
v ⊗ 3 .
These last two steps are needed in the worst case scenario where there is no gap of intensity between the of the spikes.
The experimental results suggest that it works not only for small p compared to n but also for values of p up to n .
N UMERICAL EXPERIMENTS
In this section we will investigate the empirical results of the previously mentioned applications to compare with our theoretical results.
We restricted to the dimension k = 3 for simplicity.
We give more details about the experiments settings in the appendix.
4. D ETECTION AND COMPARISON OF RECOVERY METHODS
For the detection algorithm, we use the tetrahedral graph.
We vary the dimension n , then.
The detection uses the fact that the distributions of Z ijk Z ijk and ( Z ijk + β v ⊗ 3 ) ( Z ijk + β v ⊗ 3 ) , where
Z is a random gaussian tensor and v an unitary vector, become distinct when β is large enough.
To illustrate this fact, in Fig.
4 left, we have drawn the frequency of the distributions of the melonic invariant over 500 instances of Z , in blue without signal and in orange with signal.
Thus, we use Intersection over Union as an evaluation metric to measure the accuracy of the signal detector.
[Figure]    [Figure] For the recovery algorithm in the symmetric case, we will compare our method (tetrahedral) to two type of results: (i) Other algorithmic methods: the melonic (tensor unfolding) and the homotopy.
To the best of our knowledge, they give the state of art respectively for the symmetric and asymmetric tensor (Biroli et al.
Other methods exist but are either too computationally expensive (sum of squares) or are variants of these algorithms.
(ii) Information-theoretical results: In (Richard & Montanari (2014)), it was proven that computing the global minimum v of the function v (cid:55)→
T ijk v i v j v k recovers the signal vector v 0 above a theoretical threshold β th = 2 . 87 √ n but with exponential time, and that no other approach can do signiﬁcantly better than that.
Thus, to assess the performance of the tetrahedral, we plot in red dashed line denoted ”perf” the deep minimum that is closest to v 0 , by using gradient method with an initialization in v 0 .
Interestingly, we note that not only the tetrahedral is able perform better than other methods, but is able as well to recover the spike at the theoretical threshold even at n ∼ 100 .
4. A SPIKE WITH DIFFERENT DIMENSIONS
We have in this case a spike v 1 ⊗ v 2 ⊗ v 3 and we wish to recover the three vectors.
We see in the left ﬁgure that the threshold ( n 1 n 2 n 3 ) 1 / 4 matches perfectly with the experiences when n 3 < n 1 .n 2 .
We also see in the right ﬁgure that when n 3 > n 1 .n 2 the recovery of v 3 has a different asymptotic behavior than v 1 and v 2 (it becomes n 1 / 2while it stays ( n 1 n 2 n 3 ) 1 / 4 for v 1 and v 2 ), corresponding to what our theoretical study predicted. [Figure] [Figure]
C ONCLUSION
In this paper we introduced a novel framework for the tensor PCA based on trace invariants.
Within this framework, we provide different algorithms to detect a signal vector or recover it.
These algorithms use tensor contractions that has a high potential of parallelization and computing optimization.
We illustrate the practical pertinence of our framework by presenting some examples of algorithms and prove their ability to detect and recover a signal vector linearly in time for β above the optimal algorithmic threshold.
Note that, one of these detection algorithms requires only O (1) memory requirement which could be advantageous in some applications.
Our experimental results show that the tetrahedral graph performs better than the the state of the art in the symmetric case.
Moreover, we also show that two well known algorithms (Homotopy and Tensor Unfolding) can be mapped to our framework and result to simpler graph (e.g. the melonic graph).
Interestingly, our framework is also able to extend the theoretical and practical study of tensor PCA to new and less restrictive situations like data where the dimensions of the axes are different or a sum of orthogonal spikes.
Important directions of future research is to explore the potential of more general graphs and to try to apply these new methods to real data.
A U SEFUL T HEOREMS
The following theorems are needed to the demonstration of the proposed theorems.
• Wedin perturbation : Let M = β v 0 v (cid:62)
+ N a square matrix with (cid:107) v 0 (cid:107) = 1 .
Let v denote the eigenvector with the largest eigenvalue of M .
If β > 2 (cid:107) N (cid:107) op then:
[Equation]
−
[Equation]
0 (cid:105)
[Equation]
• Chebyshev inequality : Let X be a random variable with ﬁnite expected value µ and ﬁnite non-zero variance σ 2 .
Then for any real number C > 0 ,
[Equation]
• Theorem (Gelfand’s Formula; 1941):
For any matrix norm (cid:107) .
(cid:107) , we have:
[Equation]
In particular we can choose the Frobenius norm:
[Equation]
B G AUSSIAN MEASURE
We will follow a similar approach to the one in Gurau (2017) to compute the expectations of trace invariants of a random tensor T whose components are normally distributed.
Let’s consider a graph G of order d associated to a trace invariant
I G ( T ) where T is a tensor of order k .
since the trace invariant is a contraction of pairs of indices of d copies of T , we denote the c -th index of the i -th copy of T by a ic and the set of the d indices of the i -th copy by a j D .
By the deﬁnition of the Gaussian measure we have the equation:
[Equation]
=
δ a
[Equation]
D
[Equation]
D
[Equation] [Equation]
, (9) where S ( k ) is the symmetric group (the group of all permutations of a set made of n elements, where we deﬁne a permutation of a set by an arrangement of its elements in a linear order.
For example the permutations of { 1 , 2 , 3 } are (1 , 2 , 3) , (1 , 3 , 2) , (2 , 1 , 3) , (2 , 3 , 1) , (3 , 1 , 2) , (3 , 2 , 1) .).
(cid:80) a indicates a sum over all the indices involved in the computation of the trace invariant, illustrated by a halfedge in the graph (we have k × d indices in total: k indices for each one of the d copies of T of the trace invariant expression).
δ G aa indicates that contracted indices (illustrated by being two ends of the same edge) of the tensors have to be equal.
The term δ a j D a j D τ ( j ) indicates that we set equal the c -th indices of the copy j and τ ( j ) of T for c ∈ { 1 , . . .
, d }
In order to have a clearer representation of the expression in equation 9, in particular the sum over the symmetric group, we introduce the concept of covering graph used in Gurau (2014): a covering graph of G consists in adding d/ 2 new edges of color
(also called propagators) relying pairwise the vertices of G (see Fig.
7 for an example).
We use the covering graphs to illustrate the permutations τ ∈ S ( k ) .
Then, an edge of color 0 between two vertices i and j will identify all the indices of the i -th and j -th copies of T .
Thus, it will be equivalent to the term δ a j D a j D τ ( j ) .
Using the covering graph, we will be able to rewrite equation 9.
For example, in the case where the edges could only have one color, we have the formula:
[Equation]
Where we denote E c ( G ) the set of edges of color c of G .
Thus {G (cid:48) , G (cid:48) \E 0 ( G (cid:48) )
= G} denotes the graphs which restrict to the graph G when we remove their edges of color 0 , which are by the deﬁnition the covering graphs.
Let c 1 , c 2 ∈ { 0 , . . .
, d } be two different colors of edges.
We denote F c 1 ,c 2 ( G ) the number of closed cycles (that we also call face) of 2 colors of G .
More explicitly, it consists of the number of connected subgraphs left when we keep in G only the edges of colors c 1 , c 2 (see Fig 8 for an illustration).
We denote F ( G ) ≡ (cid:80) c
1 (cid:54) =
c 2 ∈{ 1 ,...,d } .
Then, with a little work (see Gurau (2014) for a proof), we obtain :
[Equation]
= (cid:88)
[Figure]
This will be the expression we will use to calculate the expectations of our graphs if each edges has only one color.
Example: Let’s calculate the expectation of the tetrahedral graph B :
For each of the three covering graphs, there is one color having two faces (the red in Fig. 8) and two colors having only one face.
Hence the expectation of the tetrahedral graph is given by 3 n 4 :
[Equation]
(cid:88)
[Equation]
B. T HE CASE OF EDGES OF MIXED COLOR
For a general trace invariant, an edge can have two half edges of different colors.
The propagators (edges of color 0 ) still identify all the indices of their two end vertices.
So we adapt the steps in
Gurau (2014) to obtain equation 12 to this new situation.
More precisely, the cycles that we will count in a covering graph G (cid:48) of G are built by beginning from a half-edge of G then alternating until they go back to the starting half-edge by:
• go from a half-edge of G of color c 1 (cid:54) = 0 to its associated half edge of color c 2 .
• go through the propagator to the half-edge of color c 2 connected to the other end of the propagator.
[Equation]
B. T HE CASE OF A GRAPH WITH AN OPEN EDGE
We encounter this case when we want to study the statistics of a matrix built out of a trace invariant graph.
Indeed cutting an edge of color c removes the contraction of two indices and leaves them free as variables i
and i 2 .
Then in the formula (cid:16)(cid:81) 3 c =1 (cid:81) e c =( v j ,v k ) ∈E c ( G ) δ a cj ,a ck (cid:17) we replace δ a cj ,a ck by δ a cj ,i 1 δ i 2 ,a ck with i
and i 2 ﬁxed.
Thus if a face of color 0 , c contains only one of the two vertices of the edge (if the graph is connected, this only happens when we calculate the variance), its contribution is equal to one.
If it contains the two vertices, it is equal to one if i
= i 2 and zero otherwise.
B. T HE CASE OF A GRAPH WITH A MIX OF SPIKES AND RANDOM TENSORS
We encounter this case when we want to calculate the contribution of the signal to an invariant, which lead us to replace a vertex by the tensorial product of our signal vector.
Since we are interested by an expectation, and knowing that the gaussian distribution is invariant by rotation, we may as well take the signal vector as the unit vector (1 , 0 , . . . , 0) .
Then in the formula (cid:16)(cid:81)
[Equation]
=
[Equation]
(cid:81)
[Equation]
=
[Equation] [Equation]
G ) δ a cj ,a ck (cid:17) we replace δ a cj ,a ck by δ a cj ,
and we restrict the propagators to the tensors vertices.
First, we notice that if there is an odd number of spikes, the expectation will be equal to zero.
Furthermore, there is no free sum if the index identiﬁed in a cycle ends up with a contraction with a spike.
Its contribution is equal to one.
C T HE PERFECT ONE - FACTORIZATION GRAPH
In the mathematical ﬁeld of graph theory, a graph G is said to be one-factorizable if:
• It is k -regular, which means that each vertex is incident to exactly k edges.
• Its edges can be colored in k colors such that no two incident edges have the same color.
A perfect one-factorization of G happens if, for every two colors c 1 and c 2 of the k colors, restricting
G to the edges of color c 1 and c 2 leads to a connected subgraph cycle (see ﬁgure below).
It is conjectured (Wagner (1992)) that they exist for all n but, until now, they were proven only for prime numbers and up to 56 .
We use this example because of its combinatorial simplicity.
The perfect one-factorization graphs have already been studied in the context of random tensors in Ferrari et al.
[Figure]
D D ETAILS ON THE EXPERIMENTS D. D ETECTION EXPERIMENTS D.1. T HE SETTINGS D. T HE SETTINGS
For each of the recovery ﬁgures, we repeated 50 times the experiment to draw the error bar (corresponding to a 95% interval conﬁdence).
Then we plot the correlation between the vector returned by the algorithm v and the signal vector v
Each experiment is an independent instance consisting in the following steps:
• We generate randomly the n components of the signal vector v 0 and then normalize it.
• We generate randomly the n 3 components of the random tensor Z .
If we are in the symmetric case, we symmetrize it with the same normalization than Richard & Montanari (2014)):
[Equation]
• We compute the tensor T
=
Z + β v ⊗ k •
We compute our matrix associated to the invariant M constructed from T using the numpy method tensordot in Python.
• We compute its leading eigenvector v • We plot the correlation between the vector resulted from the algorithm v and the signal vector we aim to recover v 0 (namely scalar product < v , v 0 > ).
D. T HE DETECTION
We see that the detection agrees with the threshold found theoretically.
The ﬁgure in the left shows the distribution of the melonic invariant without and with noise to illustrate how the detection works .
[Figure]
    

[Figure] [Figure]
D. R ECOVERY D.4. T HE SETTINGS
[Figure]
. T HE SETTINGS
For each of the recovery ﬁgures, we repeated 50 times the experiment to draw the error bar (corresponding to a 95% interval conﬁdence).
Then we plot the correlation between the vector returned by the algorithm v and the signal vector v
Each experiment is an independent instance consisting in the following steps:
• We generate randomly the n components of the signal vector v 0 and then normalize it.
• We generate randomly the n 3 components of the random tensor Z .
If we are in the symmetric case, we symmetrize it with the same normalization than Richard & Montanari (2014)):
[Equation]
• We compute the tensor T
=
Z + β v ⊗ k •
We compute our matrix associated to the invariant M constructed from T using the numpy method tensordot in Python.
• We compute its leading eigenvector v • We plot the correlation between the vector resulted from the algorithm v and the signal vector we aim to recover v 0 (namely scalar product < v , v 0 > ).
D.5. T HE NUMBER OF ITERATIONS OF POWER ITERATION FOR EACH METHOD
[Figure]
Empirical results shows the fast convergence using the power iteration method following the tetrahedral method.
D. M EMORY AND TIME REQUIREMENTS OF THE METHODS General requirements:
Since our method consists on computing algebraic operations on tensors entries to store in matrix n × n , the naive recovery method for a tensor of rank k and for a tensor invariant of degree d would require O ( n 2 ) space requirements and O ( n kd ) time complexity.
However, similarly to the matrix case where naive implementation requires n 3 but better asymptotic bounds on the time required to multiply matrices have been known since the work of Strassen in the 1960s.
It is possible to simplify the computations by using decomposition on blocks.
Many different algorithms have been designed for multiplying matrices on different types of hardware, including parallel and distributed systems, where the computational work is spread over multiple processors (perhaps over a network).
Optimizing tensor contractions is out of the scope of this paper.
Tetrahedral method with ˜ O ( n 4 ) time and ˜ O ( n 3 ) space requirement:
We can calculate the leading eigenvector of the matrix associated to the tetrahedral: M ti 1 i 2 ≡
T
i 1 jk T i 2 j (cid:48) k (cid:48) T ij (cid:48)
k T ijk (cid:48) .
Using the decomposition, we can prove that there is an eigengap between the highest eigenvalue ∼ β 4 and the second one.
Thus, we can use matrix power method with log( n ) steps in order to recover the leading eigenvector v ← M t v .
We can compute it in a time of ˜ O ( n 4 ) and space ˜ O
( n 3 ) .
Indeed this method requires two intermediate matrices we denote A 1 , A 3 and a intermediate 3 − dimensional tensor we denote A 2 , in every step there is at most n 4 operations.
we follow these steps:
• We take an initial vector v •
We compute the matrix A 1 deﬁned as A 1 j (cid:48) k (cid:48)
≡ T i 2 j (cid:48) k (cid:48) v i• We compute the tensor
A 2 deﬁned as A 2 k (cid:48)
ik ≡ A 1 j (cid:48) k (cid:48)
T ij (cid:48)
k • We compute the matrix A 3 deﬁned as as A 3 kj ≡ A 2 k (cid:48)
ik T ijk (cid:48)
• Return the vector v (cid:48) deﬁned as v (cid:48)
i 1 ≡ A 3 kj T i 1 jk
Ours methods have in general n 2 memory requirements (where to store the matrix).
Comparison between different methods
We use our results and the results in Anandkumar et al.
(2017) to ﬁll the following table
[Table]
We didn’t ﬁll the time requirement of the tetrahedral method because we don’t know how to compute it efﬁciently.
Indeed, in the same way that there exist algorithms optimized for matrix multiplications (dividing the matrices into blocks), we use tensordot from Python for our contractions.
It uses optimized matrix multiplications to fasten drastically the calculations.
Naive implementation of the tetrahedral matrix calculation would require O ( n 6 ) , but wee see in the empirical plot that it is sharply below that complexity.
We hope that there is still a lot of optimization to be done in tensor contractions in the same way it was done for matrix calculations, especially with the rise of the use of tensorial objects.
The ability to be parallelized (each component of the matrix M G ,e can be computed independently) is also an important feature of our methods for practical applications.
[Figure]
However, to have a more accurate estimation of the time requirement, one has to take into account time to read the input (a tensor of size n 3 ).
[Figure]
E P ROOFS
Useful lemmas and deﬁnitions:
Lemma 10.
The covering graphs involved in the calculation of E (Tr( M − E ( M )) r ) are the same that are involved in the calculation of E (Tr( M r )) minus the covering graphs where one of the elementary subgraphs has only internal propagators.
Lemma 11.
If β det = O ( n 3 / 4 ) , the intermediate matrices and the intermediate graphs are never dominant.
Calculating the operational norm of a matrix The trace of the power r of a matrix Trace ( M r ) can be represented as the graph gluing open edges of 2 r subgraphs G i representing one matrix A where
M =
A (cid:62) A , we denote this graph by G .
In case we remove the expectation of the matrix, it comes back to the fact that no subgraph G i can have only internal propagators.
Elementary subgraph The subgraph corresponding to M like in Figure.
They are the building bricks.
Internal propagator : When the propagators connect vertices in the same elementary subgraph.
Proof Lemma 1 Let M be the matrix obtained by cuting the edge e of a trace invariant graph G .
We know that the equation 9 identify the indices i and j that are ends of a same edge by including a factor δ ij .
Hence, the non diagonal components M ij , i (cid:54) =
j have a null expectation since the cycle involving the edge e will identify i and j with a δ ij = 0 .
By invariance, E ( M ii ) are equal for all i .
Then, by linearity of the expectation, E ( M ii ) =
E (Tr( M )) /n
=
E ( G ) /n .
[Figure]
Proof theorem
Detecting a signal could be done through calculating the value at T of the invariant associated to G and comparing it to the pure noise distribution of this invariant.
Let n ∈ N and G a graph of degree d .
When β → ∞ , the dominant graph will be the one where all the tensors are replaced by the signal spike, with a value of β d .
The subdominant ones will be the graphs where all but one tensors vertices have been replaced by a spike.
Thus it is easy to show that E ( I G ( T ))
→ β d while σ ( I G ( T )) ∼ β ( d − 1) .
Then, using Chebyshev’s inequality demonstrates that the distributions of I G ( T ) , β → ∞ and I G ( T ) , β
= 0 are completely disjoint which allows the detection of a signal.
Thus, ∃ β det such that ∀ β ≥ β det , the detection is possible.
[Figure]
[Figure]
Proof theorem 3 Let n ∈ N , G a graph of degree d and M the matrix obtained by cutting the edge e .
When β → ∞ for a ﬁxed n , the dominant matrix by operator norm will be the one where all the vertices are replaced by signal spike, and which is proportional to vv T .
Then applying Wedin perturbation theorem proves that ∃ β rec such that ∀ β ≥ β rec , the recovery is possible.
Proof for theorem 4 To show that (cid:13)(cid:13) M ( N ) − E ( M ( N ) )
(cid:13)(cid:13) op ≥ n dk/ 4 , it is enough to see that the variance of any graph of degree d is superior than dk/ 4 .
Take two copies of a graph G and connect corresponding points from each copy to the other.
The number of cycles will be equal to dk/ 2 .
And the variance will be at least equal to n dk/ 4 (See example below).
Since the expectation is equal to the sum on covering graphs, the variance will always be larger than n dk/ 4 .
As for the variance of
Tr (cid:0) ( MM T ) r (cid:1) it will be always superior to 2 rd .
c y c l ec y c l ecycle
Proof of theorem 5 : We will use the lemma 10.
Let’s prove by induction that E (Tr (cid:0) ( MM T ) r (cid:1) ) ≤ n kr +1 where M is the matrix associated to the melon by opening one of its edge.
For r = 1 it is straightforward by simply enumerating the cycles.
Now, let r > 1 and let’s assume the statement is true for all r (cid:48) <
r .
Let’s consider one of its elementary subgraphs (deﬁned in the beginning of the section).
Using lemma 10, we can divide the edges of the subgraph in two sets:
• k − 1 edges that are in faces of at least two edges.
• One external edge divided in half edges.
If the k − 1 edges are in faces of exactly two edges, itt means that the propagators at both the left and right vertices are connected to the vertices of the same other elementary subgraph.
This will lead to two possibilities:
• The two elementary subgraphs are next to each other.
Then we remove these two elementary subgraphs and connect the two left open edges.
We will have a contribution of n k from the two elementary subgraphs and use the induction hypothesis to show that the remaining graph has a contribution of maximum n k ( r − 1)+1 .
Combining them we have a total contribution of n k ( r − 1)+1+ k = n kr +1 which proves the theorem.
• The two subgraphs are not next to each other: Then the open edges have also a contribution of maximum 1 / 2 .
Cutting this elementary subgraph and using the induction hypothesis proves the theorem.
Thus using Gelfand’s Formula, the norm operator is equal to n 3 / 2 .
Which gives an recovery threshold for the melon of β = n 3 / 4 using Wedin perturbation.
For the symmetric case, since T symijk = (cid:80) τ ∈ S k T τ ( i ) τ ( j ) τ ( k ) where S k is the symmetric group introduced previously.
We just need to expand the melonic matrix T symi 1 jk T symi 2 jk in function of the asymmetric tensor T .
We obtain a sum of nine matrices (corresponding to graphs with mixed colors).
We can show that the operational norm of each of these matrices is at most equal to n 3 / 2 .
Proof of theorem 6 : Using the matrix decomposition illustrated in 18, we can see that we have a sum of four matrices of rank one.
Using the usual formula to calculate the expectation of the operational norm of the matrices we obtain:
• (cid:13)(cid:13) M
( N ) (cid:13)(cid:13) =
n• (cid:13)(cid:13) M
( D ) (cid:13)(cid:13) =
n • (cid:13)(cid:13) M ( R ) (cid:13)(cid:13) = n 3 /
we can see that the ﬁrst one dominates by n 1 / 4 the last one (consisting in the signal vector) so we are able to use power iteration or one of its variants (like in Anandkumar et al.
(2017) and Biroli et al.
to recover it.
[Figure]
[Equation]
Proof theorem 7 : Let’s deﬁne a set of even numbers ( s i ) i ∈{ 0 ,...,r } where s i / 2 is equal to the number of propagators having its two vertices in the subgraph G i (we call these propagators internal).
Thus we have s/ 2 ≡ =
(cid:80) i s i / 2 internal propagators in total.
We can divide the edges of the subgraph G in three classes.
Each edge could be part of a face of either one edge, two edges, three edges, etc.. Let’s count how many we can have of each type in an optimal conﬁguration (maximizing the number of faces) for each class.
• Edges only connected to internal propagators, there is at most (cid:80)
i s i ( s i − 1) / 2 such edges.
– We can have a maximum of s i / 2 faces of one edge since that is the number of internal edges.
– Then, from the s i ( s i − 2) / 2 remaining ones, if we want to put them in faces of two edges, the two edges needs too to be in the same subgraph.
Thus, we can create a maximum of s i / 2( s i / 2 − 1) /
= s i
( s i − 2) / 8 different couples of propagators.
Each couple can’t create more than one face since otherwise it will contradict the maximally single trace hypothesis of the graph.
So there is a maximum of s i ( s i − 2) / 4 edges that can be put in two edges faces.
– The remaining s i ( s i − 2) / 4 will be in faces of at least three edges.
• Edges connected to one internal and one external propagators, there is at most (cid:80)
i s i ( k + 1 − s i ) one such edges.
They can only be in faces of at least three edges.
• Edges only connected to external propagators, there is at most (cid:80) i (( k + 1 − s i )
( k − s i ) / 2) such edges.
They can only be put in faces of at least two edges.
• The open edge can be put in a face of maximum one edge.
So we have a maximal number of faces given by:
[Equation]
Let’s focus in the case where s i = 0 .
If all the points of the graph are connected to points of another graph, then the two extremes of the open edge connects to the same graph.
In that case the open edge has a contribution of only 1 / 2 .
In the other case, then there is at least k edges which contributes only by at most 1 / 3 making them lose 1 / 6 .
Either way the contribution of the subgraph can’t go further then k ( k + 1) / 4 .
We obtain:
[Equation]
We can easily check that this quantity is always smaller than k ( k +1) for k > 5 .
Proof of theorem 8 : Let’s consider the melonic graph with the edge associated to the index of the space of dimension n l open.
It is associated to the matrix:
[Equation]
For simplicity the graphs will be drawn for k = 3 .
We will treat two separate cases:
• Case n l ≤ (cid:89)
[Equation]
Let’s prove by induction the statement:
[Equation]
– Initialisation with r = 2 : For M − E ( M ) we have the following graph and its covering graphs
which gives
[Equation]
– Assume S r is true for all r (cid:48) <
r , let’s prove it for r .
* In the case where there is no cycle of color l of length 1 (which means a propagator that shares the same ends that an edge of color l ), then all edges have a contribution of 1 / 2 at maximum.
So the contribution of that covering graph will be at most equal to (cid:81) ki
=1 n r/ 2 i .
In case there is such a propagator, which means that there is a propagator coinciding with an edge of color l between two elementary subgraphs.
Let r
Then either the two other ends of the two elementary subgraphs are connected by a propagator like in 22 or not.
In the ﬁrst case
the total number of cycles will be at most equal to one cycle of each color, plus the number of cycles of a graph with
r − 1)
elementary subgraphs, so E (Tr (cid:0) M 2 r (cid:1) ≤ 2 n l (cid:81)
ki =1 n ri .
of of
and
subgraphs adjacent to
hypothesis
[Figure]
Let’s prove by induction the statement: S r : E
[Tr(( M − E ( M )) r )]
= O ( n rl (cid:81) i (cid:54) =
l n i ) .
– Initialisation with r = 2 : For M − E ( M ) we have the covering graphs in 19 which checks S– Assume S r is true for all r (cid:48) <
r , let’s prove it for r .
* In the case where there is no cycle of color l of length 1 (which means a propagator that shares the same ends that an edge of color l ), then all edges have a contribution of 1 / 2 at maximum.
So the contribution of that covering graph will be at most equal to (cid:81) ki
=1 n r/ 2 i .
In case there is such a propagator, which means that there is a propagator coinciding with an edge of color l between two elementary subgraphs.
Let r
Then either the two other ends of the two elementary subgraphs are connected by a propagator like in 22 or not.
In the ﬁrst case, the total number of cycles will be at most equal to one cycle of each color, plus the number of cycles of a graph with 2( r − 1)
[Figure]
[Equation]
[Figure]
Proof of theorem 9 : Let ˜ A
= M e G ( T ) the matrix we have with noise and A = (cid:80) pi =1 β di v ki the matrix when there is no noise and only signal.
We call an eigenspace of A the subspace of R n spanned by some eigenvectors of A .
The idea of the proof of the theorem is that for β large enough, the eigenspace spanned by the signal vectors S should be close to the eigenspace spanned by the p largest eigenvectors of ˜ A denoted ˜ S .
Then, we take a vector from this subspace ˜ S , we perform power iteration on it in order to recover a signal vector w 1 , and repeat this step by restricting to the subspace of ˜ S orthogonal to w 1 .
If there were only one signal, there is a simple way to calculate how close are v 1 and w 1 : θ ( v 1 , w 1 ) =
cos − 1 ( v 1 .w 1 ) .
To quantify the closeness of two subspaces S and ˜ S , we use the canonical angles.
The ﬁrst canonical angle is deﬁned as follow:
[Equation]
[Equation]
where we denoted the maximizers by x 1 and y 1 and x.y denotes the scalar product between x and y .
Then we deﬁne by iteration the k th canonical angle by:
[Equation]
We put these angles in a matrix Θ
[Equation]
We deﬁne now the distance between S and ˜ S by (cid:107) sin(Θ)
(cid:107) F with sin(Θ) deﬁned as:
[Equation]
The Davis-Kahan Theorem uses these tools to provide a bound on subpaces spanned by eigenvectors under perturbation (Stewart & guang Sun (1990)):
Davis-Kahan: Let A , A (cid:48) two symmetric matrices with respective eigenvectors:
[Equation]
Let’s deﬁne the eigengap as:
[Equation]
Then if δ >
[Equation]
The result also holds for (cid:107) .
(cid:107) op
Let’s study the case where β d > cn k/ 4 where c is a constant large enough.
The gap will be equal to δ =
( c − 1) n k/ 4 .
In the case of the melonic graph, (cid:13)(cid:13)(cid:13) ˜
A − A (cid:13)(cid:13)(cid:13) op can be bounded by 2 n k/ 4 .
Let’s take a vector z from ˜ S .
It can be put in the form of:
[Equation]
where q (cid:48) and q are unitary vectors, and q = q (cid:48) − (cid:80) pi
=1 ( q (cid:48) .
x k ) x
k
so it is orthogonal to all the vectors v i and that |
γ 0 | ≤ |
α 0 | .
Let’s put a high bound for alpha 0 .
Since x k .y k
= cos( θ k ) , ∀ k ∈ { 1 , . . .
, p }
[Equation]
Using these results, we show that power iteration on a vector z of S is able to recover a signal vector v k :
[Equation]
Thus, repeating z ← T zz will converge to v l max with l max = max( { β l γ 2 l , l ∈ { 1 , . . .
, r }} )
We repeat this step p times, but before each step we use deﬂation by computing:
[Equation]
Proof Lemma
The covering graphs involved in the calculation of E (Tr( M − E ( M )) r ) are the same that are involved in the calculation of E (Tr( M r )) minus the covering graphs where one of the elementary subgraphs has only internal propagators.
We can ﬁrst begin by showing that if we have two matrices M 1 of degree d 1 and M 2 of degree d 2 , then the covering graphs involved in Tr(( M 1 − E ( M 1 ))
M 2 ) are the covering graphs of
Tr( M 1 M 2 ) minus the ones where the propagators are internal.
Let’s denote the color of the open edge by l .
To distinguish the d 1 + d 2 vertices and their indices, we assign numbers to each vertex.
Namely the vertices of M 1 are denoted by 1 , . . .
, d 1 with 1 and 2 associated to the two vertices adjacent to the open edge of M 1 , and d 1 , . . .
, d 1 + d 2 where the vertices of the open edge of Mare d 1 + 1 , d 1 + 2 .
We denote all the indices by a , and the indices of all colors associated to a vertex l by a D l .
A subset of the symmetric group σ ( d 1 + d 2 ) are the permutations which could be written as τ 1 τwhere τ 1 is a permutation of (1 , . . .
, d 1 ) and τ 2 is a permutation of ( d 1 , . . .
, d 1 + d 2 ) .
We denote by the symbol δ B aa the δ of all the indices that are contracted in the graph.
Let B 1 the graph associated to M 1 and B 2 the graph associated to M 2 .
Let B (cid:48) the subgraph of B obtained by opening an edge of color c corresponding to the vertices i and j , it is an elementary subgraph of B r , the graph corresponding to Tr( M r ) .
We have
[Equation]
Since the edges of B are the union of the edges of B 1 and B 1 and an edge of color l connecting the vertices of the open edges: 1 with d 1 +1 and 2 with d 1 +2 , we have:
[Equation]
This let us write:
[Equation]
Thus the only permutations that are not cancelled are the permutations that does not live invariant the subset { 1 , . . .
, d 1 } , which means that ∃ i ∈ { 1 , . . .
, d 1 } such that τ ( i ) ∈ { d 1 + 1 , . . .
, d 1 + d 2 } .
These permutations correspond to the covering graphs that does not have only internal propagators connecting the vertices of M 1 .
This proves our claim.
[Figure]
Proof of lemma 11 : Let G a graph that has β dec = n k/ 4 as a detection threshold.
Let’s assume that an intermediate graph G in with r spikes dominate for β > β dec and let’s take the covering graph of G in 2 which gives the contribution for the domination.
We cut the the propagators that involves the spikes and connect them together, while we connect by a new propagator each vertex of spike to its mirror.
This covering graph will give a strictly superior contribution to the expectation if the graph is connected since:
• The cycles that didn’t involve a spike stay unchanged •
The cycles that involved a spike adds together a new contribution of n α with α > 3 r/ 4 .
Thus for n 3 / 4 << β << n α , the pure noise graph dominates the intermediate graph, which leads to a contradiction.
The demonstration for the matrices M follows the same steps but instead of G 2 , one has to consider Tr( M r ) .