Sparse Hypergraph Community Detection Thresholds in Stochastic Block Model
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Community detection in random graphs or hypergraphs is an interesting fundamental problem in statistics, machine learning and computer vision.
When thehypergraphs are generated by a stochastic block model , the existence of a sharpthreshold on the model parameters for community detection was conjectured by
Angelini et al. 2015.
In this paper, we conﬁrm the positive part of the conjecture,the possibility of non-trivial reconstruction above the threshold, for the case of twoblocks by comparing the hypergraph stochastic block model with its Erdös-Rényicounterpart.
Furthermore, we show the negative part of the conjecture by relatingthe model with the so-called multi-type Galton-Watson hypertrees and consideringthe broadcasting problem on these hypertrees.
The methods developed in this paperare generalised from the study of sparse random graphs by Mossel et al.
Introduction
Community detection, or clustering, aims to identify groups sharing similar properties from a globalpopulation.
It is a fundamentally important problem in statistics, network analysis and computervision [15, 36, 11, 31, 27].
Many clustering algorithms have been developed based on graphs inwhich each edge contains exactly two vertices representing pairwise relationships on data.
However,real-world data are often much more complicated than the ordinary graph structures.
For example, inthe co-authorship network [9, 25, 28], the number of co-authors may vary from paper to paper so thatone can hardly study edges consisting of two co-authors only.
To represent higher order relationships among data, hypergraphs have been proposed to model dataand have been shown to have some advantages over graphs [38].
In the co-authorship example above,a generalised edge (called hyperedge ) allows us to consider the connectivity of arbitrarily many coauthors.
In recent years, hypergraphs have been widely used to model complex data relationships, forinstance, in robust multi-structure clustering [27], bioinformatics [34], and social networks extraction[37].
However, from both theoretical and methodological sides, the complexity of hypergraphscreates much more challenges in the sense that many analysis tools developed in the graph case cannot be easily generalised to the hypergraph case.
In general, analysing the spectral properties of theadjacency tensor of a hypergraph is more difﬁcult than studying the spectrum of the adjacency matrixof a graph.
The stochastic block model (SBM), also known as the planted partition model , was initially proposedby Holland et al.
[17] to model random graphs with community structures.
The general idea behindthis model is to label each vertex as a community member identically and independently, and thenconstruct a random graph based on the membership and some given probabilities.
It becomes thewell-known Erdös-Rényi graph model if we ignore the membership restriction.
The recovery ordetection problem, based on the SBM, is to ﬁnd the community membership when the number ofvertices is sufﬁciently large.
SBM often serves as a benchmark for clustering algorithms on graph data.
Likewise, an analogous model called the hypergraph stochastic block model (HSBM) wasproposed to model higher-order relationships for random hypergraphs [13].
In what follows, we willpresent a brief introduction to hypergraphs, HSBM and its associated detection problem.
1.1 The stochastic block model for hypergraphs
A hypergraph is a pair H =
( V , E ) , where V is a set of vertices and E is a set of hyperedges , which isa collection of subsets of V .
The degree of a vertex v 2 V is the number of hyperedges in H thatcontains v .
The degree of a hyperedge e 2 E is the number of vertices contained in e .
The hypergraph
H is called d -uniform if the degree of every hyperedge e 2 E is d .
A 2 -uniform hypergraph isa graph.
An l -cycle in H is a cyclic ordering of a subset of the vertex set, and of hyperedges,where any two adjacent hyperedges have exactly l common vertices.
We call an l -cycle loose ifl = 1 .
For instance, let H
= ( V , E ) be a 3 -uniform hypergraph, where V = { v 1 , v 2 , · · · , v 7 } ,
E = { ( v i , v j , v k ) | 1  i < j < k  7 } .
Then { ( v 1 , v 2 , v 3 ) , ( v 3 , v 4 , v 5 ) , ( v 5 , v 6 , v 1 ) } is a loose cycle.
There are many ways to represent a hypergraph.
For instance, the incidence matrix of a hypergraph H is deﬁned as A 2 { 0 , 1 } |V| ⇥ |E| , where A ( v, e ) = 1 if the vertex v 2 V is contained in the 50 hyperedge e 2 E , and 0 otherwise.
The adjacency tensor of H is deﬁned as T , where T i 1 , ··· ,i d = 1 51 if { i 1 , · · · , i d } is a hyperedge of H , and 0 otherwise.
The adjacency matrix of H is deﬁned as 52 B =
( B ij ) |V| ⇥ |V| , where B ij =
P e : { i,j } 2 e T e if i 6 = j , and 0 otherwise, here e is a hyperedge and
T e is an element in its associated adjacency tensor.
Unfortunately, most of the computations involvingthe adjacency tensor are NP-hard, for examples, determining the feasibility of a system of bilinearequations, rank-1 approximation of a tensor, whether a tensor has a given spectral norm [16].
For simplicity, from now on, we specify our d -uniform hypergraph stochastic block model with tworoughly balanced communities.
Considering non-uniform hypergraphs or hypergraphs with multicommunities may require extra effort, which will be reserved for future work.
Let H =
([ n ] , E ) be a duniform hypergraph with vertex set [ n ] := { 1 , 2 , · · · , n } and hyperedge set E ,  :=
(  1 , · · · ,  n ) =
{ +1 ,  1 } n the spins on [ n ] , which means each vertex i 2 [ n ] is assigned with a spin  i 2 { +1 ,  1 } .
61 Let
S n be the set of all pairs ( H,  ) , we can generate a random pair ( H,  ) from the ﬁnite set S n asfollows ([13, 14, 20, 26]):
• First generate i.i.d random variables  i 2 { +1 ,  1 } uniformly for each i 2 [ n ] .
• Then, for the obtained
= (  1 , · · · ,  n ) , we generate a random d -uniform hypergraphH where an hyperedge e
= { i 1 , · · · , i d } is included independently with probability p n
if  i 1 = · · · =  i d , and with probability q n otherwise, where 0 < q n < p n < 1 ( p n , q n
possibly depending on n ).
We denote by H d ( n, p n , q n ) a d -uniform hypergraph generated by the process above.
(With a littlenotation abuse, we sometimes denote the distribution that such hypergraphs follow by the samenotation).
In particular, H 2 ( n, p n , q n ) is a random graph generated by the graph version of the SBM.
Suppose C 1 = { i 2 [ n ] |  i = +1 } and C
= { i 2 [ n ] |  i =
1 } are two communities in the 72 hypergraph H .
The goal of community detection is to estimate the unknown spin  up to a signﬂip by observing H only from a sample ( H,  ) drawn from H d ( n, p n , q n ) , i.e., to ﬁnd estimatorsˆ  i 2 { +1 ,  1 } that are correlated with the true partition  i .
By correlation, we mean there exists a 75 constant c 2 (0 , 1) , for any ✏ > 0 ,
[Equation]
where O n (ˆ  ,  ) :
= 1 n
P i 2 [ n ]  i ˆ  i is the empirical overlap between  and ˆ  .
Recall the deﬁnitions of exact recovery and almost exact recovery of communities in the literature(for example, see [26]).
By exact recovery , we mean ﬁnding a spin vector ˆ  2 { +1 ,  1 } n such that
[Equation]
That is, the estimated spin vector ˆ  is either exactly the true spin vector  or exactly the negative truespin vector   .
By almost exact recovery , we mean ﬁnding a spin vector ˆ  2 { +1 ,  1 } n such that, 81 for any ✏ > 0 ,
[Equation]
That is, the empirical overlap O n (ˆ  ,  ) is in [  1 ,  1 + ✏ ) [ (1  ✏ , 1] asymptotically almost surely.
This case almost exactly recovers the community, as indicated by the terminology, compared with thedeﬁnition of exact recovery.
Note that we only consider the detection (not (almost) exact recovery) ofcommunities in this paper.
The study of sparse graphs/hypergraphs as well as those with constant average degree, is wellmotivated from the viewpoint of real networks.
For instance, many large social networks like Twitteror LinkedIn, may have several millions of nodes but only ten times more edges [21].
A coauthorshipnetwork may have millions of nodes, but the average degree is no more than 20 in general [25].
Therefore, It is natural to consider the sparse HSBM with p n and q n of the order O ⇣ 1 /  nd [Equation] ⌘ .
Decelle et al.
[7] conjectured the existence of a sharp threshold, called Kesten-Stigun threshold , ofdetection in the sparse graph SBM based on some ideas from statistical physics.
More precisely, ifp n = an , q n = bn , where a, b are constants independent of n , then detection is possible if and only if( a  b ) 2 > 2( a + b ) .
This conjecture was conﬁrmed in [22, 23, 24, 6], for examples, by studying thespectra of the non-backtracking [6], self-avoiding [22] or graph powering [1] matrices.
Moreover,some efﬁcient algorithms were proposed to achieve detection at the threshold [6, 32, 1].
In the case of the hypergraph SBM, Angelini et al.
[2] conjectured the existence of a sharp threshold onthe model parameters, for community detection in the sparse hypergraph.
If p n = [Equation] i.e., the hypergraph H ⇠ H d ( n, p n , q n ) has constant expected vertex degree.
Pal et al.
[26] provedthe positive part of the conjecture, the detection is possible if  2 > ↵ , where ↵ =
( d  1) a +(2 d  1  1) b 2 d , =
( d  1) a  b 2 d  1 .
Their method is based on the spectral analysis of the so-called self-avoiding matrixassociated with the hypergraph, which is a generalisation to random hypergraphs of the methoddeveloped by Massoulié [22] for sparse random graphs.
Moreover, the consistent estimators formodel parameters a and b are still unknown.
In the present paper, we reconﬁrm the positive part of the conjecture by comparing the sparse HSBMwith an appropriate Erdös-Rényi hypergraph model (see Theorem 1.1).
Further, in the same theoremwe give consistent estimators for a and b if the detection is possible.
More importantly, we showthat it is impossible (see Theorem 1.3) to detect communities if  2 < ↵ by coupling our hypergraphwith the multi-type Galton-Watson hypertree and studying broadcasting problems on hypertree.
Thethreshold of broadcasting on a (hyper)tree results in that of detection on the HSBM.
Our methods aremotivated by the study of the community detection problem on the graph SBM conducted by Mosselet al.
The main results of the present paper are summarised as follows.
1.2 Main results
Let H d ⇣ n, p n +(2 d  1  1) q n 2 d ⌘ be the Erdös-Rényi model in which each hyperedge is included with acommon probability p n +(2 d  1  1) q n 2 d , which is chosen to make sure this model has the same averagedegree as H d ( n, p n , q n ) .
In other words, H d
[Equation]
⌘ represents a d -uniform randomhypergraph without any community structures.
In particular, H n, p n + q [Equation]
is the traditional ErdösRényi random graph that has been extensively studied in the graph literature [4, 12, 8].
Let P n and ˜ P n denote the probability measures with respect to H d ( n, p n , q n ) and
H d ⇣ n, p n +(2 d  1  1) q n 2 d ⌘ , respectively.
Then P n and ˜ P n are said to be mutually contiguous if for anymeasurable sets A n , P n ( A n ) !
if and only if ˜ P n ( A n ) !
as n ! 1 .
P n and ˜ P n are said to be 122 asymptotically orthogonal if there exists a sequence of measurable sets A n such that P n ( A n ) !
0and ˜ P n ( A n ) !
1 as n ! 1 .
From now on, we ﬁx the following notation,
[Equation]
where a > b > 0 are some constants independent of n , d  2 is an integer.
Deﬁne
[Equation]
where ↵ measures the expected degree of any vertex and  measures the discrepancy between thenumber of neighbors with different signs of any vertex.
Theorem 1.1.
If  2 > ↵ , then P n and ˜ P n are asymptotically orthogonal.
Let X ⇣ n be the number ofloose cycles of length ⇣ n and deﬁne
[Equation]
where ⇣ n = b log 1 / 4 n c , |E| is the number of observed hyperedges, then ˆ a n =
[Equation]
and ˆ
[Equation]
[Equation] ⌘ are consistent estimators for a and b , respectively.
Namely, ˆ a n !
a and ˆ b n !
b in probability as n ! 1 .
Theorem 1.2.
If  2 < ↵ , then for any ﬁxed vertices v 1 and v 2 , H ⇠ H d ( n, p n , q n ) ,
[Equation]
Theorem 1.3.
If  2 < ↵ , then P n and ˜ P n are mutually contiguous.
Further more, there is noconsistent estimator for a and b if  2 < ↵ .
When d = 2 , the condition  2 = ↵ becomes ( a  b ) 2 = 2( a + b ) , which is the threshold for communitydetection in the (graph-based) SBM.
Readers may refer to [22, 23, 24] for more detail.
Intuitivelyspeaking, Theorem 1.1 says that H d ( n, p n , q n ) and H d
[Equation]
⌘ are asymptoticallyalmost surely distinguishable above the threshold (i.e., can distinguish a community with structurefrom a random hypergraph with the same average degree).
While Theorem 1.3 states that they arenot distinguishable below the threshold.
Figure 1 shows the phase transition of community detectionin the sparse HSBM
where d = 3 , 4 , 5 .
Consider any two random vertices of a hypergraph H ,
Theorem 1.2 claims that no algorithm can tell whether they have the same spin, i.e., the probability ofidentifying them with the same spin is asymptotically almost surely the same as the probability ofﬁnding them with different spins.
Put loosely, it is not possible to do any better than random chancewhen assigning labels.
[Figure] [Equation]
[Figure] [Equation]
[Equation]
[Figure]
b
[Equation]
[Figure]
[Equation]
[Figure]
[Equation] [Equation]
[Equation]
The remainder of the present paper is organised as follows:
In section 2, we study the probabilitydistribution of the number of ⇣ n -loose cycle X ⇣ n and then prove Theorem 1.1.
In section 3, based onthe multi-type Galton-Watson hypertree proposed in [26], we investigate the connection between thedetection problem on H d ( n, p n , q n ) and the broadcasting problem on the hypertree, which providessome tools to prove Theorem 1.2.
Mossel et al.
in [23] adopted Theorem 4.1 in [35] that gavesufﬁcient conditions for contiguity to prove Theorem 1.3 where d = 2 (i.e., the graph SBM case).
Toprove Theorem 1.3 in the sparse HSBM case, we need to verify all conditions of Theorem 4.1 in [35].
Section 4 concludes the paper, including some further discussion.
In this section, we mainly show that the number of ⇣ n -loose cycles of H ⇠ H d ( n, p n , q n ) or
H
[Equation]
⌘ is asymptotically approximately Poisson-distributed.
In generalinhomogeneous graphs, Bollobás et al.
[5] demonstrated that the number of cycles asymptoticallyfollows the Poisson distribution.
We borrow their proof techniques to get a similar conclusion in thehypergraph case.
Then, by comparing the ﬁrst two moments of the number of ⇣ n -loose cycles, wecan ﬁnd that H d ( n, p n , q n ) and H d ⇣ n,
[Equation]
⌘ are asymptotically orthogonal if  2 > ↵ .
Theorem 2.1.
Let X ⇣ n be the number of ⇣ n -loose cycle of a hypergraph H .
Suppose ⇣ n =
O ⇣ log 1 / 4 ( n ) ⌘ .
(1) If H ⇠ H d ( n, p n , q n ) , then X ⇣ n d !
Pois ⇣ ↵ ⇣ n +  ⇣ n 2 ⇣ n ⌘ ;
(2) If H ⇠ H d ⇣ n, p n +(2 d  1  1) q n 2 d ⌘ , then X ⇣ n d !
Pois
[Equation]
n 2 ⇣ n ⌘ ,
where d !
denotes convergent in distribution.
For any non-negative integer s , let [ X ] s be the product
X ( X  1) · · · ( X  s + 1) .
By Lemma2.8 in [35] (Lemma A.1 in the Appendix), if we can prove the expectation E
[ X ⇣ n ] r !
µ r , then
X ⇣ n d !
Pois ( µ ) .
Refer to the Appendix for the detailed proof.
With the preparation provided by establishing Theorem 2.1, now we can prove our ﬁrst mainresult, Theorem 1.1.
We use P n ( · ) , E ( · ) , Var( · ) to denote probability, mean and variation withrespect to H d ( n, p n , q n ) while notations with tilde above denote their counterparts with respect to
H d ⇣ n, p n + [Equation] ⌘ .
Proof of Theorem 1.1.
Since  2 > ↵ , there exists a constant ⇢ satisfying  > ⇢ > p ↵ .
Then
E X ⇣ n = Var X ⇣ n = o ( ⇢ 2 ⇣ n ) and ˜ E
X ⇣ n = ˜Var
X ⇣ n = o ( ⇢ 2 ⇣ n ) as n ! 1 .
Let A n := { X ⇣ n  ˜ E
X ⇣ n + ⇢ ⇣ n } , by Chebyshev’s inequality,
[Equation]
Since E X ⇣ n  ˜
E X ⇣ n ⇠  ⇣ n 2 ⇣ n = !
( ⇢ ⇣ n ) , then for larger enough ⇣ n , E X ⇣ n  ⇢
⇣ n  ˜ E
X ⇣ n + ⇢ ⇣ n ,which implies P n ( A n )  P n ( X ⇣ n  E X ⇣ n  ⇢ ⇣ n ) .
By Chebyshev’s inequality,
[Equation]
By (1) and (2), we have shown that P n and ˜ P n are asymptotically orthogonal.
Now we show that Theorem 2.1 can provide consistent estimators for a and b .
First of all, we have aconsistent estimator ˆ ↵ n for ↵ by simply counting the number of hyperedges.
If we can prove that can be consistently estimated by ˆ  n =  2 ⇣ n X ⇣ n  ˆ ↵ ⇣ n n  1 ⇣ n , then ˆ a n = 1 d ⇣ ˆ ↵ n + (2 d  1  1)ˆ  n ⌘ and ˆ b n = 1 d ⇣ ˆ ↵ n  ˆ  n ⌘ are consistent estimators for a and b , respectively.
Take ⇢ 2 ( p ↵ ,  ) , by Chebyshev’s inequality,
[Equation]
[Equation] [Equation]
p
Var X
[Equation]
 ⇢ ⇣ n 2 ⇣ n p
Var X ⇣ n
A  1  2 ⇣ n p
Var X ⇣ n ⇢ ⇣ n ! !
1 , as ⇣ n ! 1 .
Thus, 2 ⇣ n X ⇣ n  ↵ ⇣ n 2 [  ⇣ n  ⇢ ⇣ n ,  ⇣ n + ⇢ ⇣ n ] a.a.s.
Since ⇢ ⇣ n = o (  ⇣ n ) as ⇣ n ! 1 , we have2 ⇣ n X ⇣ n  ↵ ⇣ n = (1 + o (1))
⇣ n a.a.s.
Since ˆ ↵ ⇣ n n !
↵ ⇣ n as ⇣ n ! 1 , we have 2 ⇣ n X ⇣ n  ˆ ↵ ⇣ n n =(
+ o (1))
⇣ n a.a.s.
Therefore, ˆ  n = (2 ⇣ n X ⇣ n  ˆ ↵ ⇣ n n ) 1 ⇣ n !
as ⇣ n ! 1 .
Non-reconstruction of the community structures
To prove Theorem 1.2, we use a multi-type Galton-Watson hypertree (proposed in [26]) that iscoupled with H d ( n, p n , q n ) .
Intuitively speaking, a neighborhood of a vertex in H d ( n, p n , q n ) lookslike the hypertree with high probability.
We take advantage of the threshold of broadcasting on thehypertree to show the threshold for community detection on the HSBM.
Let T be an inﬁnite rooted hypertree with root ⇢ constructed as follows:
Generate a root ⇢ withspin ⌧ ⇢ 2 { +1 ,  1 } and then generate Pois( ↵ ) many hyperedges that only intersects at ⇢ .
For 195 s  d  1 , a hyperedge is called type s if there are s many children (i.e., the vertices in thehyperedge) with spin ⌧ ⇢ and d  1  s many children with spin  ⌧ ⇢ in the hyperedge.
Hyperedgesof type d  1 are generated with probability ( d  1) a ↵ 2 d  1 and of type s with probability ( d  1) b ( d  1 s ) ↵ 2 d for0  s  d  2 .
It is easy to check that
[Equation]
We generate hyperedges of different types i.i.d.
by Poisson distribution.
For each hyperedge of types , in the ﬁrst generation, we uniformly and randomly pick s vertices with spin ⌧ ⇢ and the rest ofthe d  1  s vertices with spin  ⌧ ⇢ .
The subsequent generations are generated by induction.
The 202 generated random hypertree, denoted by ( T, ⇢ , ⌧ ) , is called a multi-type Galton-Watson hypertree .
Figure 2 shows an example of a multi-type Galton-Watson hypertree where d = 3 .
The broadcasting problem on ( T, ⇢ , ⌧ ) is whether the spin ⌧ ⇢ could be deduced from the spins atgeneration l of the hypertree, where l is sufﬁciently large.
Such a problem on general inﬁnite treeswas initially proposed by Kesten and Stigum in 1966 [19], although they used somewhat differentterminologies.
This problem was solved by Blecher et al.
[3] and Evans et al.
Let T l be the vertices of generations no more than l in T , @
T l the vertices of generation l in T , thenwe have (refer to the Appendix for the proof):
Theorem 3.1.
Suppose ( T, ⇢ , ⌧ ) is a multi-type Galton-Watson hypertree where the offspring distribution has mean ↵ > 1 , if  2 < ↵ , then
[Equation]
where ⌧ @
T l = { ⌧ v | v 2 @
T l } .
Let ( H, ⇢ ) be a rooted hypergraph, i.e., a hypergraph H with a distinguished vertex ⇢ 2 V ( H ) .
Tworooted hypergraphs ( H 1 , ⇢ 1 ) and ( H 2 , ⇢ 2 ) are said to be isomorphic if and only if there is a bijection' :
V ( H 1 ) !
V ( H 2 ) such that ' ( ⇢ 1 ) = ⇢ 2 and e 2 E ( H 1 )
if and only if ' ( e ) := { ' ( i ) | i 2 e }
E ( H 2 ) .
Let ( H, ⇢ ,  ) be a rooted hypergraph with a spin vector  , i.e., each vertex
i 2 V ( H ) is givena spin  ( i ) 2 { +1 ,  1 } .
Two rooted hypergarphs ( H 1 , ⇢ 1 ,  1 ) and ( H 2 , ⇢ 2 ,  2 ) are said to bespin-preserving isomorphic , denoted by ( H 1 , ⇢ 1 ,  1 ) ⌘
( H 2 , ⇢ 2 ,  2 ) , if and only if there is anisomorphism ' : ( H 1 , ⇢ 1 ) !
( H 2 , ⇢ 2 ) such that  1 ( j ) =
2 ( ' ( j )) for any j 2 V ( H 1 ) .
Let ( H, ⇢ ,  ) l be the rooted hypergraph ( H, ⇢ ,  ) truncated at distance l from ⇢ and ( T, ⇢ , ⌧ ) lthe
rooted hypertree ( T, ⇢ , ⌧ ) truncated at generation (distance) l from ⇢ .
If l = c log( n ) withc log( ↵ ) < 1 / 4 and c is a constant, then for sufﬁciently large n (see Theorem 5.2 in [26]),(
) if  ⇢ = +1 , there exists a coupling between ( H, ⇢ ,  ) and ( T, ⇢ , ⌧ ) such that
[Equation]
(2) if  ⇢ =
1 , there exists a coupling between ( H, ⇢ ,  ) and ( T, ⇢ ,  ⌧ ) such that
[Equation]
Namely, we have
[Equation]
which means a neighborhood in H looks like a broadcasting process on a multi-type Galton-Watsonhypertree T .
The statement of Theorem 1.2 is equivalent to E (  ⇢ | H,  v ) ! 0 , and also equivalent to 230 Var(  ⇢ | H,  v ) !
In order to prove Theorem 1.2, we can consider the variance Var(  ⇢ | H,  @ H l ) ,which asymptotically almost surely is equal to Var( ⌧ ⇢ | T, ⌧ @ T l ) .
By the property of monotonicity ofvariance, we can add conditions on the spins  v and  @
H l , which can make the reconstruction easier.
Then we wish to remove the condition  v and such action will not lose much information, i.e.,  @
H lgives much more information in the process.
To ﬁll the gap that once we have conditioned on  @
H l ,we can remove the condition on  v , the following lemma is required.
Lemma 3.2.
Let V 1 , V 2 , V 3 be a random partition of the vertex set V ( H ) such that V 2 separates
V 1 and V 3 (i.e., any path connecting vertices in V 1 and V 3 must pass through vertices in V 2 ).
If| V 1 [ V 2 |
= o ( p n ) for a.a.e H , then
[Equation]
for a.a.e.
H and  .
Now we can establish the second main result, Theorem 1.2.
Proof of Theorem 1.2.
By the property of the monotonicity of conditional variance, for v / 2 H l , wehave
[Equation]
By Lemma 3.2,  ⇢ and  v are a.a.s.
conditionally independent given  @
H l and H , whichmeans Var(  ⇢ | H,  v ,  @
H l ) !
Var(  ⇢ | H,  @ H l ) .
By Theorem 5.2 in [26], Var(  ⇢ | H,  @
H l ) !
245 Var( ⌧ ⇢ | T, ⌧ @ T l ) .
By Theorem 3.1, Var( ⌧ ⇢ | T, ⌧ @ T l ) !
1 a.a.s.
Therefore, Var(  ⇢ | H,  v ) !
1 246 a.a.s.
In the rest of this section, we will show our last main result, Theorem 1.3.
The second half of Theorem1.3 can be derived from its ﬁrst half as follows:
If  2 < ↵ , suppose ˆ a and ˆ b are estimators for a and bwhich converge in probability, we claim their limits depend on a + (2 d  1  1) b only.
To verify this,let A, B be another choice of the model parameters with ( d  1)
( A  B ) 2 < 2 d  1 ( A +(2 d  1  1) B )and A + (2 d  1  1) B = a + (2 d  1  1) b .
Let P 0 n be the probability with respect to the model
H d ✓ n, A ( nd  1 ) , B ( nd  1 ) ◆ , and suppose
[Equation]
For any ✏ > 0 , let A n be the event that | ˆ a  a ⇤ | > ✏ , then P n ( A n ) ! 0 .
By the ﬁrst half of Theorem1.3, ˜ P n ( A n ) ! 0 .
Since A + (2 d  1  1) B = a + (2 d  1  1) b , using the ﬁrst half of Theorem 1.3again, we have P 0 n ( A n ) ! 0 , which implies A ⇤ =
a ⇤ .
Therefore, ˆ a converges to the same limitunder P n and P 0 n , i.e., ˆ a is not a consistent estimator for a (same argument for b ).
The ﬁrst half of Theorem 1.3 is a bit difﬁcult to prove.
Here, we give some outline of how to establishit, and the lengthy proof is put in the Appendix.
For our purpose, we will use the small subgraphconditioning method, which was initially proposed by Robinson et al.
[29, 30] and then applied by
Janson [18] to prove contiguity.
The follow theorem from [35] is a crucial tool to show the contiguitybetween H d ( n, p n , q n ) and H d
[Equation]
⌘ when  2 < ↵ .
Theorem 3.3 (Theorem 4.1 in [35]) .
Let P n and ˜ P n be two ﬁxed sequences of probability measureson a common measurable space, Y n = P n ˜ P n the density of P n with respect to ˜ P n .
For i  1 , let  i > 0 , i   1 , for each n , suppose random variables X in satisfy
(C1) X in d !
W i as n !
1 jointly for all i under ˜ P n , where W i
⇠ Pois
(  i ) are independent
Poisson variables
(C2)
For every non-negative integers
[Equation]
[Equation]
[Equation]
[Equation]
Then, ˜ P n and P n are contiguous.
In order to apply the theorem above, we will take Y n to be the likelihood ratio of the adjacencytensor A of H , and X in to be the number of i -loose cycles in H .
To shorten notation, we adopti 1 ⇤ i d to represent the ordering i 1 i 2 · · · i d .
A i
⇤ i d :=
A
i 1 i 2 ··· i d .
I [  i
⇤  i d ] :=
I [  i 1 = · · · =  i d ] , 274 which is 1 if  i 1 = · · · =  i d , and 0 otherwise.
% ( s, t ) =
{ ( i 1 , · · · , i s ) | 1  i 1  · · ·  i s  t } .ˆ d :
[Equation]
2 d . p 0 :
= ˆ d ( nd  1 ) , q 0 :
= 1  p 0 .
Then, for H ⇠ H d ( n, p n , q n ) , its hyperedgeprobabilities p i
⇤ i
d
(  ) and q i
⇤ i
d
(  ) can be given by
[Equation]
⇤
[Equation]
For H ⇠ H d ( n, p 0 ) , its hyperedge probabilities can be given by P ( { i 1 ⇤ i d } ) =
p A
i
⇤ idq 1  A
i
⇤ id.
Therefore, the likelihood ratio Y n = P n ( A )
˜ P n ( A ) of the adjacency tensor A can be written as
[Equation]
Obviously, we have ˜ E Y n = 1 .
The condition (C1) is easy to verify as we can see from Theorem2.1, which also implies that  i can be chosen as ↵ i 2 i .
Then, we need to estimate the asymptotics of
˜ E ( Y n [ X 1 n ] s 1 · · · [ X kn ] s k ) and ˜ E Y 2 n , where the calculation of ﬁrst moment is somewhat similar tocalculations in the proof of Theorem 2.1.
The following lemma gives an estimation for the secondmoment of Y n .
Lemma 3.4.
[Equation]
After verifying all the conditions of Theorem 3.3 (see the Appendix) the ﬁrst half of Theorem 1.3follows directly.
Conclusion
Community detection is of great interest in many ﬁelds such as network analysis, statistics andcomputer vision.
Hypergraphs have been widely used to model higher-order relationships amongdata.
The SBM (graph-based) model is popular in community detection but only directly modelspair-wise interactions.
Its hypergraph version, HSBM, likewise serves as a benchmark for clusteringalgorithms on higher order interaction (hypergraph) data.
This paper provides theoretical analyses on the community detection thresholds in the sparse HSBM.
Speciﬁcally, we address a previous conjecture that there exists a sharp threshold on the modelparameters for community detection to be possible.
We proved that above the threshold, it is possibleto asymptotically almost surely construct a bi-partition of the hypergraph positively correlated with thetrue bi-partition.
Moreover, below the threshold, we show that community detection is informationtheoretically impossible.
Importantly, we also provide consistent estimators for the model parameters,when the detection is possible.
Our estimators require the counting of the number of loose cycles in the hypergraph.
To our bestknowledge, there is no efﬁcient algorithm to count the number of loose cycles in hypergraphs ingeneral.
Therefore, from the algorithmic side, it might be more practical to count the number ofnon-backtracking/self-avoiding walks of the same length [26, 33], which can be computed by matrixmultiplication efﬁciently.
Nevertheless, from the theoretical side, conﬁrming the positive side of thethreshold conjecture by analysing the number of loose cycles has not been done before.
Moreover,providing more practical estimators (or examining whether one can estimate the number of loosecycles sufﬁciently accurate to still use the estimators of model parameters we provide), is a worthytopic of future research.
Note that the non-uniform hypergraphs can be viewed as a ‘superposition’ of a collection of uniformhypergraphs.
Thus it may be possible to generalise our proofs to cover non-uniform hypergraphsin future work.
Considering other variants of the hypergraph community detection problem, forinstances, multi-community structures, extremely sparse scenario ( p n , q n ⌧ 1 /n d  1 ) , moderatesparse scenario ( p n , q n =
O (log( n ) /n
d  1 )) , dense scenario ( p n , q n  1 /n d  1 ) , are future researchdirections that would complete the work presented here.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes] See the abstract and introduction.(b) Did you describe the limitations of your work?
[Yes] See the last two paragraphs of theconclusion section.(c)
Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes] We have read the ethics review guidelines.2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See the mainresults subsection.(b)
Did you include complete proofs of all theoretical results?
[Yes] Some proofs are added
in the Appendix.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[N/A]
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]