Sparse Hypergraph Community Detection Thresholds in Stochastic Block Model
Abstract
Community detection in random graphs or hypergraphs is an interesting fundamental problem in statistics, machine learning and computer vision.
When the hypergraphs are generated by a stochastic block model , the existence of a sharp threshold on the model parameters for community detection was conjectured by
Angelini et al. 2015.
In this paper, we conﬁrm the positive part of the conjecture, the possibility of non-trivial reconstruction above the threshold, for the case of two blocks. We do so by comparing the hypergraph stochastic block model with its Erdös-Rényi counterpart.
Furthermore, we show the negative part of the conjecture by relating the model with the so-called multi-type Galton-Watson hypertrees and considering the broadcasting problem on these hypertrees.
The methods developed in this paper are generalised from the study of sparse random graphs by Mossel et al. 2015.
Introduction
Community detection, or clustering, aims to identify groups sharing similar properties from a global population.
It is a fundamentally important problem in statistics, network analysis and computer vision [15, 36, 11, 31, 27].
Many clustering algorithms have been developed based on graphs in which each edge contains exactly two vertices representing pairwise relationships on data.
However, real-world data are often much more complicated than the ordinary graph structures.
For example, in the co-authorship network [9, 25, 28], the number of co-authors may vary from paper to paper so that one can hardly study edges consisting of two co-authors only.
To represent higher order relationships among data hypergraphs have been proposed to model data and have been shown to have some advantages over graphs [38].
In the co-authorship example above, a generalised edge (called hyperedge ) allows us to consider the connectivity of arbitrarily many coauthors.
In recent years, hypergraphs have been widely used to model complex data relationships, for instance, in robust multi-structure clustering [27], bioinformatics [34], and social networks extraction [37].
However, from both theoretical and methodological sides, the complexity of hypergraphs creates much more challenges in the sense that many analysis tools developed in the graph case can not be easily generalised to the hypergraph case.
In general, analysing the spectral properties of the adjacency tensor of a hypergraph is more difﬁcult than studying the spectrum of the adjacency matrix of a graph.
The stochastic block model (SBM), also known as the planted partition model , was initially proposed by Holland et al.
[17] to model random graphs with community structures.
The general idea behind this model is to label each vertex as a community member identically and independently, and then construct a random graph based on the membership and some given probabilities.
It becomes the well-known Erdös-Rényi graph model if we ignore the membership restriction.
The recovery or detection problem, based on the SBM, is to ﬁnd the community membership when the number of vertices is sufﬁciently large.
SBM often serves as a benchmark for clustering algorithms on graph data.
Likewise, an analogous model called the hypergraph stochastic block model (HSBM) was proposed to model higher-order relationships for random hypergraphs [13].
In what follows, we will present a brief introduction to hypergraphs, HSBM and its associated detection problem.
1.1 The stochastic block model for hypergraphs
A hypergraph is a pair H =
( V , E ) , where V is a set of vertices and E is a set of hyperedges , which is a collection of subsets of V .
The degree of a vertex v 2 V is the number of hyperedges in H that contains v .
The degree of a hyperedge e 2 E is the number of vertices contained in e .
The hypergraph
H is called d -uniform if the degree of every hyperedge e 2 E is d .
A 2 -uniform hypergraph is a graph.
An l -cycle in H is a cyclic ordering of a subset of the vertex set, and of hyperedges, where any two adjacent hyperedges have exactly l common vertices.
We call an l -cycle loose if l = 1 .
For instance, let H
= ( V , E ) be a 3 -uniform hypergraph, where V = { v 1 , v 2 , · · · , v 7 } ,
E = { ( v i , v j , v k ) | 1  i < j < k  7 } .
Then { ( v 1 , v 2 , v 3 ) , ( v 3 , v 4 , v 5 ) , ( v 5 , v 6 , v 1 ) } is a loose cycle.
There are many ways to represent a hypergraph.
For instance, the incidence matrix of a hypergraph
H is deﬁned as A 2 { 0 , 1 } |V| ⇥ |E| , where A ( v, e ) = 1 if the vertex v 2 V is contained in the hyperedge e 2 E , and 0 otherwise.
The adjacency tensor of H is deﬁned as T , where T i 1 , ··· ,i d = 1 if { i 1 , · · · , i d } is a hyperedge of H , and 0 otherwise.
The adjacency matrix of H is deﬁned as B =
( B ij ) |V| ⇥ |V| , where B ij =
P e : { i,j } 2 e T e if i 6 = j , and 0 otherwise, here e is a hyperedge and
T e is an element in its associated adjacency tensor.
Unfortunately, most of the computations involving the adjacency tensor are NP-hard, for examples, determining the rank-1 approximation of a tensor, whether a tensor has a given spectral norm [16].
For simplicity, from now on, we specify our d -uniform hypergraph stochastic block model with two roughly balanced communities.
Considering non-uniform hypergraphs or hypergraphs with multicommunities may require extra effort, which will be reserved for future work.
Let H =
([ n ] , E ) be a duniform hypergraph with vertex set [ n ] := { 1 , 2 , · · · , n } and hyperedge set E ,  :=
(  1 , · · · ,  n ) =
{ +1 ,  1 } n the spins on [ n ] , which means each vertex i 2 [ n ] is assigned with a spin  i 2 { +1 ,  1 } .
Let
S n be the set of all pairs ( H,  ) , we can generate a random pair ( H,  ) from the ﬁnite set S n as follows ([13, 14, 20, 26]):
• First generate i.i.d random variables  i 2 { +1 ,  1 } uniformly for each i 2 [ n ] .
• Then, for the obtained
= (  1 , · · · ,  n ) , we generate a random d -uniform hypergraph H where an hyperedge e
= { i 1 , · · · , i d } is included independently with probability p n if  i 1 = · · · =  i d , and with probability q n otherwise, where 0 < q n < p n < 1 ( p n , q n possibly depending on n )
We denote by H d ( n, p n , q n ) a d -uniform hypergraph generated by the process above.
(With a little notation abuse, we sometimes denote the distribution that such hypergraphs follow by the same notation).
In particular, H 2 ( n, p n , q n ) is a random graph generated by the graph version of the SBM.
Suppose C 1 = { i 2 [ n ] |  i = +1 } and C
= { i 2 [ n ] |  i =
1 } are two communities in the hypergraph H .
The goal of community detection is to estimate the unknown spin  up to a sign ﬂip by observing H only from a sample ( H,  ) drawn from H d ( n, p n , q n ) , i.e., to ﬁnd estimators ˆ  i 2 { +1 ,  1 } that are correlated with the true partition  i .
By correlation, we mean there exists a constant c 2 (0 , 1) , for any ✏ > 0 ,
[Equation]
where O n (ˆ  ,  ) :
= 1 n
P i 2 [ n ]  i ˆ  i is the empirical overlap between  and ˆ  .
Recall the deﬁnitions of exact recovery and almost exact recovery of communities in the literature (for example, see [26]).
By exact recovery , we mean ﬁnding a spin vector ˆ  2 { [Equation] 1 ,  1 } n such that
[Equation]
That is, the estimated spin vector ˆ  is either exactly the true spin vector  or exactly the negative true spin vector   .
By almost exact recovery , we mean ﬁnding a spin vector ˆ  2 { [Equation] 1 ,  1 } n such that, for any ✏ > 0 ,
[Equation]
That is, the empirical overlap O n (ˆ  ,  ) is in [  1 ,  1 + ✏ ) [ (1  ✏ , 1] asymptotically almost surely.
This case almost exactly recovers the community, as indicated by the terminology, compared with the deﬁnition of exact recovery.
Note that we only consider the detection (not (almost) exact recovery) of communities in this paper.
The study of sparse graphs/hypergraphs as well as those with constant average degree, is well motivated from the viewpoint of real networks.
For instance, many large social networks like Twitter or LinkedIn, may have several millions of nodes but only ten times more edges [21].
A coauthorship network may have millions of nodes, but the average degree is no more than 20 in general [25].
Therefore, It is natural to consider the sparse HSBM with p n and q n of the order O
[Equation]
Decelle et al.
[7] conjectured the existence of a sharp threshold, called Kesten-Stigun threshold , of detection in the sparse graph SBM based on some ideas from statistical physics.
More precisely, if p n = an , q n = bn , where a, b are constants independent of n , then detection is possible if and only if ( a  b ) 2 > 2( a + b ) .
This conjecture was conﬁrmed in [22, 23, 24, 6], for examples, by studying the spectra of the non-backtracking [6], self-avoiding [22] or graph powering [1] matrices.
Moreover, some efﬁcient algorithms were proposed to achieve detection at the threshold [6, 32, 1].
In the case of the hypergraph SBM, Angelini et al.
[2] conjectured the existence of a sharp threshold on the model parameters, for community detection in the sparse hypergraph.
If p n = a ( nd  1 ) , q n = b ( nd  1 ) , i.e., the hypergraph H ⇠ H d ( n, p n , q n ) has constant expected vertex degree.
Pal et al.
[26] proved the positive part of the conjecture, the detection is possible if  2 > ↵ , where ↵ =
( d  1) a +(2 d  1  1) b 2 d ,  = ( d  1) a  b 2 d  1 .
Their method is based on the spectral analysis of the so-called self-avoiding matrix associated with the hypergraph, which is a generalisation to random hypergraphs of the method developed by Massoulié [22] for sparse random graphs.
Moreover, the consistent estimators for model parameters a and b are still unknown.
In the present paper, we reconﬁrm the positive part of the conjecture by comparing the sparse HSBM with an appropriate Erdös-Rényi hypergraph model (see Theorem 1.1).
Further, in the same theorem we give consistent estimators for a and b if the detection is possible.
More importantly, we show that it is impossible (see Theorem 1.3) to detect communities if  2 < ↵ by coupling our hypergraph with the multi-type Galton-Watson hypertree and studying broadcasting problems on hypertree.
The threshold of broadcasting on a (hyper)tree results in that of detection on the HSBM.
Our methods are motivated by the study of the community detection problem on the graph SBM conducted by Mossel et al.
The challenges in generalizing the results to hypergraphs and the way how we overcame them are summarized at the end of the supplementary material.
The main results of the present paper are summarised as follows.
1.2 Main results
Let H d
[Equation]
n
[Equation] ⌘ be the Erdös-Rényi model in which each hyperedge is included with a common probability p n + [Equation] d
[Equation]
, which is chosen to make sure this model has the same average degree as H d ( n, p n , q n ) .
In other words, H d
[Equation]
n,
[Equation]
⌘ represents a d -uniform random hypergraph without any community structures.
In particular, H
[Equation]
is the traditional ErdösRényi random graph that has been extensively studied in the graph literature [4, 12, 8].
Let P n and ˜ P n denote the probability measures with respect to H d ( n, p n , q n ) and H d ⇣ n,
[Equation] + [Equation] ⌘ , respectively.
Then P n and ˜ P n are said to be mutually contiguous if for any measurable sets A n , P n ( A n ) !
if and only if ˜ P n ( A n ) !
as n ! 1 .
P n and ˜ P n are said to be asymptotically orthogonal if there exists a sequence of measurable sets A n such that P n ( A n ) ! {+
+}
and ˜ P n ( A n ) !
1 as n ! 1 .
From now on, we ﬁx the following notation,
[Equation]
where a > b > 0 are some constants independent of n , d  2 is an integer.
Deﬁne
[Equation]
where ↵ measures the expected degree of any vertex and  measures the discrepancy between the number of neighbors with different signs of any vertex.
Theorem 1.1.
If  2 > ↵ , then P n and ˜ P n are asymptotically orthogonal.
Let X ⇣ n be the number of loose cycles of length ⇣ n and deﬁne
[Equation]
where ⇣ n = b log 1 / 4 n c , |E| is the number of observed hyperedges, then ˆ a n
[Equation]
[Equation]
and ˆ
[Equation]
⌘ are consistent estimators for a and b , respectively.
Namely, ˆ a n !
a and ˆ b n !
b in probability as n ! 1 .
Theorem 1.2.
If  2 < ↵ , then for any ﬁxed vertices v 1 and v 2 , H ⇠ H d ( n, p n , q n ) ,
[Equation]
Theorem 1.3.
If  2 < ↵ , then P n and ˜ P n are mutually contiguous.
Further more, there is no consistent estimator for a and b if  2 < ↵ .
When d = 2 , the condition  2 = ↵ becomes ( a  b ) 2 = 2( a + b ) , which is the threshold for community detection in the (graph-based) SBM.
Readers may refer to [22, 23, 24] for more detail.
Intuitively speaking, Theorem 1.1 says that H d ( n, p n , q n ) and H d
[Equation]
[Equation] ⌘ are asymptotically almost surely distinguishable above the threshold (i.e., can distinguish a community with structure from a random hypergraph with the same average degree).
While Theorem 1.3 states that they are not distinguishable below the threshold.
Figure 1 shows the phase transition of community detection in the sparse HSBM where d = 3 , 4 , 5 .
Consider any two random vertices of a hypergraph H ,
Theorem 1.2 claims that no algorithm can tell whether they have the same spin, i.e., the probability of identifying them with the same spin is asymptotically almost surely the same as the probability of ﬁnding them with different spins.
Put loosely, it is not possible to do any better than random chance when assigning labels.
The remainder of the present paper is organised as follows:
In section 2, we study the probability distribution of the number of ⇣ n -loose cycle X ⇣ n and then prove Theorem 1.1.
In section 3, based on the multi-type Galton-Watson hypertree proposed in [26], we investigate the connection between the detection problem on H d ( n, p n , q n ) and the broadcasting problem on the hypertree, which provides some tools to prove Theorem 1.2.
Mossel et al.
in [23] applied Theorem 4.1 in [35] that gave sufﬁcient conditions for contiguity to prove Theorem 1.3 where d = 2 (i.e., the graph SBM case).
To prove Theorem 1.3 in the sparse HSBM case, we need to verify all conditions of Theorem 4.1 in [35].
Section 4 concludes the paper, including some further discussion.
The distribution of loose cycles
In this section, we mainly show that the number of ⇣ n -loose cycles of H ⇠ H d ( n, p n , q n ) or H ⇠ H d ⇣ n, p n
[Equation]
⌘ is asymptotically approximately Poisson-distributed.
In general inhomogeneous graphs, Bollobás et al.
[5] demonstrated that the number of cycles asymptotically follows the Poisson distribution.
We borrow their proof techniques to get a similar conclusion in the hypergraph case.
Then, by comparing the ﬁrst two moments of the number of ⇣ n -loose cycles, we can ﬁnd that H d ( n, p n , q n ) and H d ⇣
[Equation]
⌘ are asymptotically orthogonal if  2 > ↵ .
10 a     b d=2d=3d= a=b
[Figure]
r
[Figure]
⇣
[Figure] ⌘ [Figure] Theorem 2.1.
Let X ⇣ n be the number of ⇣ n -loose cycle of a hypergraph H .
Suppose ⇣ n =
O ⇣ log 1 / 4 ( n ) ⌘ .
(1) If H ⇠ H d ( n, p n , q n ) , then X ⇣ n d !
Pois [Equation]
(2) If H ⇠ H d ⇣ n, p n +(2 d  1  1) q n 2 d  [Equation] ⌘ , then X ⇣ n d !
Pois
[Equation]
⇣
[Equation]
where d !
denotes convergent in distribution.
For any non-negative integer s , let [ X ] s be the product
X ( X  1) · · · ( X  s + 1) .
By Lemma 2.8 in [35] (Lemma A.1 in the Appendix), if we can prove the expectation E [ X ⇣ n ] r ! µ r , then
X ⇣ n d !
Pois ( µ ) .
Refer to the Appendix for the detailed proof.
With the preparation provided by establishing Theorem 2.1, now we can prove our ﬁrst main result, Theorem 1.1.
We use P n ( · ) , E ( · ) , Var( · ) to denote probability, mean and variation with respect to H d ( n, p n , q n ) while notations with tilde above denote their counterparts with respect to
H d ⇣
[Equation]
(2 d  1  1) q n 2 d ⌘ .
Proof of Theorem 1.1.
Since  2 > ↵ , there exists a constant ⇢ satisfying  > ⇢ > p ↵ .
Then
E X ⇣ n = Var X ⇣ n = o ( ⇢ 2 ⇣ n ) and ˜ E X ⇣ n = ˜
Var X ⇣ n = o ( ⇢ 2 ⇣ n ) as n ! 1 .
Let A n := { X ⇣ n  ˜ E
X ⇣ n + ⇢ ⇣ n } , by Chebyshev’s inequality,
[Equation]
Since E X ⇣ n  ˜
E X ⇣ n ⇠  ⇣ n 2 ⇣ n = !
( ⇢ ⇣ n ) , then for larger enough ⇣ n , E X ⇣ n  ⇢
⇣ n  ˜ E
X ⇣ n + ⇢ ⇣ n , which implies P n ( A n )  P n ( X ⇣ n  E X ⇣ n  ⇢ ⇣ n ) .
By Chebyshev’s inequality,
[Equation]
By (1) and (2), we have shown that P n and ˜ P n are asymptotically orthogonal.
Now we show that Theorem 2.1 can provide consistent estimators for a and b .
First of all, we have a consistent estimator ˆ ↵ n for ↵ by simply counting the number of hyperedges.
If we can prove that  can be consistently estimated by ˆ  n =  [Equation] ⇣
[Equation] n  1 ⇣ n , then ˆ a n = 1 d
[Equation]
⇣ ˆ ↵
[Equation]
+ (2 d  1  1)ˆ  n ⌘ and ˆ b n = 1 d ⇣ ˆ ↵ n  ˆ  n ⌘ are consistent estimators for a and b , respectively.
Take ⇢ 2 ( p ↵ ,  ) , by Chebyshev’s inequality, [Equation]
Thus, 2 ⇣ n X ⇣ n  ↵ ⇣ n 2 [  ⇣ n  ⇢ ⇣ n ,  ⇣ n + ⇢ ⇣ n ] a.a.s.
Since ⇢ ⇣ n = o (  ⇣ n ) as ⇣ n ! 1 , we have 2 ⇣ n X ⇣ n  ↵ ⇣ n = (1 + o (1))
⇣ n a.a.s.
Since ˆ ↵ ⇣ n n !
↵ ⇣ n as ⇣ n ! 1 , we have 2 ⇣ n X ⇣ n  ˆ ↵ ⇣ n n [-=(
-]= (1 + o (1))
⇣ n a.a.s.
Therefore, ˆ  n = (2 ⇣ n X ⇣ n  ˆ ↵ ⇣ n n ) 1 ⇣ n !
as ⇣ n ! 1 .
Non-reconstruction of the community structures
To prove Theorem 1.2, we use a multi-type Galton-Watson hypertree (proposed in [26]) that is coupled with H d ( n, p n , q n ) .
Intuitively speaking, a neighborhood of a vertex in H d ( n, p n , q n ) looks like the hypertree with high probability.
We take advantage of the threshold of broadcasting on the hypertree to show the threshold for community detection on the HSBM.
Let T be an inﬁnite rooted hypertree with root ⇢ constructed as follows:
Generate a root ⇢ with spin ⌧ ⇢ 2 { +1 ,  1 } and then generate Pois( ↵ ) many hyperedges that only intersects at ⇢ .
For s  d  1 , a hyperedge is called type s if there are s many children (i.e., the vertices in the hyperedge) with spin ⌧ ⇢ and d  1  s many children with spin  ⌧ ⇢ in the hyperedge.
Hyperedges of type d  1 are generated with probability ( d  1) a ↵ 2 d  1 and of type s with probability ( d  1) b ( d  1 s ) ↵ 2 d {+for 
+}  s  d  2 .
It is easy to check that
[Equation]
We generate hyperedges of different types i.i.d.
by Poisson distribution.
For each hyperedge of type s , in the ﬁrst generation, we uniformly and randomly pick s vertices with spin ⌧ ⇢ and the rest of the d  1  s vertices with spin  ⌧ ⇢ .
The subsequent generations are generated by induction.
The generated random hypertree, denoted by ( T, ⇢ , ⌧ ) , is called a multi-type Galton-Watson hypertree .
Figure 2 shows an example of a multi-type Galton-Watson hypertree where d = 3 .
The broadcasting problem on ( T, ⇢ , ⌧ ) is whether the spin ⌧ ⇢ could be deduced from the spins at generation l of the hypertree, where l is sufﬁciently large.
Such a problem on general inﬁnite trees was initially proposed by Kesten and Stigum in 1966 [19], although they used somewhat different terminologies.
This problem was solved by Blecher et al.
[3] and Evans et al.
Let T l be the vertices of generations no more than l in T , @
T l the vertices of generation l in T , then we have (refer to the Appendix for the proof):
Theorem 3.1.
Suppose ( T, ⇢ , ⌧ ) is a multi-type Galton-Watson hypertree where the offspring distribution has mean ↵ > 1 , if  2 < ↵ , then
[Equation]
where ⌧ @ T l = { ⌧ v | v 2 @
T l } .
Let ( H, ⇢ ) be a rooted hypergraph, i.e., a hypergraph H with a distinguished vertex ⇢ 2 V ( H ) .
Two rooted hypergraphs ( H 1 , ⇢ 1 ) and ( H 2 , ⇢ 2 ) are said to be isomorphic if and only if there is a bijection ' : V ( H 1 ) !
V ( H 2 ) such that ' ( ⇢ 1 ) = ⇢ 2 and e 2 E ( H 1 )
if and only if ' ( e ) := { ' ( i ) | i 2 e }[-
-]
E ( H 2 ) .
Let ( H, ⇢ ,  ) be a rooted hypergraph with a spin vector  , i.e., each vertex
i 2 V ( H ) is given a spin  ( i ) 2 { +1 ,  1 } .
Two rooted hypergarphs ( H 1 , ⇢ 1 ,  1 ) and ( H 2 , ⇢ 2 ,  2 ) are said to be spin-preserving isomorphic , denoted by ( H 1 , ⇢ 1 ,  1 ) ⌘
( H 2 , ⇢ 2 ,  2 ) , if and only if there is an isomorphism ' : ( H 1 , ⇢ 1 ) !
( H 2 , ⇢ 2 ) such that  1 ( j ) =
2 ( ' ( j )) for any j 2 V ( H 1 ) .
Let ( H, ⇢ ,  ) l be the rooted hypergraph ( H, ⇢ ,  ) truncated at distance l from ⇢ and ( T, ⇢ , ⌧ ) l the rooted hypertree ( T, ⇢ , ⌧ ) truncated at generation (distance) l from ⇢ .
If l = c log( n ) with c log( ↵ ) < 1 / 4 and c is a constant, then for sufﬁciently large n (see Theorem 5.2 in [26]),
(1) if  ⇢ = +1 , there exists a coupling between ( H, ⇢ ,  ) and ( T, ⇢ , ⌧ ) such that
[Equation]
(2) if  ⇢ =
1 , there exists a coupling between ( H, ⇢ ,  ) and ( T, ⇢ ,  ⌧ ) such that
[Equation]
Namely, we have
[Equation]
which means a neighborhood in H looks like a broadcasting process on a multi-type Galton-Watson hypertree T .
The statement of Theorem 1.2 is equivalent to E (  ⇢ | H,  v ) ! 0 , and also equivalent to Var(  ⇢ | H,  v ) !
In order to prove Theorem 1.2, we can consider the variance Var(  ⇢ | H,  @ H l ) , which asymptotically almost surely is equal to Var( ⌧ ⇢ | T, ⌧ @ T l ) .
By the property of monotonicity of variance, we can add conditions on the spins  v and  @
H l , which can make the reconstruction easier.
Then we wish to remove the condition  v and such action will not lose much information, i.e.,  @
H l gives much more information in the process.
To ﬁll the gap that once we have conditioned on  @
H l , we can remove the condition on  v , the following lemma is required.
Lemma 3.2.
Let V 1 , V 2 , V 3 be a random partition of the vertex set V ( H ) such that V 2 separates
V 1 and V 3 (i.e., any path connecting vertices in V 1 and V 3 must pass through vertices in V 2 ).
If | V 1 [ V 2 |
= o ( p n ) for a.a.e H , then
[Equation]
for a.a.e.
H and  .
Now we can establish the second main result, Theorem 1.2.
Proof of Theorem 1.2.
By the property of the monotonicity of conditional variance, for v / 2 H l , we have
[Equation]
| H,  v ,  @ H l )  Var(  ⇢ | H,  v ) .
[Equation]
By Lemma 3.2,  ⇢ and  v are a.a.s.
conditionally independent given  @
H l and H , which means Var(  ⇢ | H,  v ,  @
H l ) !
Var(  ⇢ | H,  @ H l ) .
By Theorem 5.2 in [26], Var(  ⇢ | H,  @
H l ) !
Var( ⌧ ⇢ | T, ⌧ @ T l ) .
By Theorem 3.1, Var( ⌧ ⇢ | T, ⌧ @ T l ) !
1 a.a.s.
Therefore, Var(  ⇢ | H,  v ) !
1 a.a.s.
In the rest of this section, we will show our last main result, Theorem 1.3.
The second half of Theorem 1.3 can be derived from its ﬁrst half as follows:
If  2 < ↵ , suppose ˆ a and ˆ b are estimators for a and b which converge in probability, we claim their limits depend on a + (2 d  1  1) b only.
To verify this, let A, B be another choice of the model parameters with ( d  1)
( A  B ) 2 < 2 d  1 ( A +(2 d  1  1) B ) and A + (2 d  1  1) B = a + (2 d  1  1) b .
Let P 0 n be the probability with respect to the model
H d ✓ n,
[Equation]
( nd  [Equation] ) , B ( nd  1 ) ◆ , and suppose
[Equation]
For any ✏ > 0 , let A n be the event that | ˆ a  a ⇤ | > ✏ , then P n ( A n ) ! 0 .
By the ﬁrst half of Theorem 1.3, ˜ P n ( A n ) ! 0 .
Since A + (2 d  1  1) B = a + (2 d  1  1) b , using the ﬁrst half of Theorem 1. again, we have P 0 n ( A n ) ! 0 , which implies A ⇤ =
a ⇤ .
Therefore, ˆ a converges to the same limit under P n and P 0 n , i.e., ˆ a is not a consistent estimator for a (same argument for b ).
The ﬁrst half of Theorem 1.3 is a bit difﬁcult to prove.
Here, we give some outline of how to establish it, and the lengthy proof is put in the Appendix.
For our purpose, we will use the small subgraph conditioning method, which was initially proposed by Robinson et al.
[29, 30] and then applied by
Janson [18] to prove contiguity.
The follow theorem from [35] is a crucial tool to show the contiguity between H d ( n, p n , q n ) and H d
[Equation]
Theorem 3.3 (Theorem 4.1 in [35]) .
Let P n and ˜ P n be two ﬁxed sequences of probability measures on a common measurable space, Y n = P n ˜ P n the density of P n with respect to ˜ P n .
For i  1 , let  i > 0 ,  i   1 , for each n , suppose random variables X in satisfy
(C1) X in d !
W i as n !
1 jointly for all i under ˜ P n , where W i ⇠ Pois (  i ) are independent Poisson variables; (C2)
For every non-negative integers s 1 , · · · , s k ,
[Equation]
(C3) P i  1  i  2 i < 1 ; (C4) ˜
E Y 2 n / (˜ E Y n ) 2 !
exp( P i  1  i  2 i ) .
Then, ˜ P n and P n are contiguous.
In order to apply the theorem above, we will take Y n to be the likelihood ratio of the adjacency tensor A of H , and X in to be the number of i -loose cycles in H .
To shorten notation, we {+adopt i 
+} ⇤ i d to represent the ordering i 1 i 2 · · · i d .
A i
⇤ i d :=
A
i 1 i 2 ··· i d .
I [  i
⇤  i d ] :=
I [  i 1 = · · · =  i d ] , which is 1 if  i 1 = · · · =  i d , and 0 otherwise.
% ( s, t ) =
{ ( i 1 , · · · , i s ) | 1  i 1  · · ·  i s  t } .
ˆ d := a +(2 d  1  1) b 2 d . p 0 :
= ˆ d ( nd  1 ) , q 0 :
= 1  p 0 .
Then, for H ⇠ H d ( n, p n , q n ) , its hyperedge probabilities p i
⇤ i
d
(  ) and q i
⇤ i
d
(  ) can be given by
[Equation]
For H ⇠ H d ( n, p 0 ) , its hyperedge probabilities can be given by P ( { i 1 ⇤ i d } ) =
p A
i
⇤ idq 1  A
i
⇤ id.
Therefore, the likelihood ratio Y n = P n ( A )
˜ P n ( A ) of the adjacency tensor A can be written as
[Equation]
Obviously, we have ˜ E Y n = 1 .
The condition (C1) is easy to verify as we can see from Theorem 2.1, which also implies that  i can be chosen as ↵ i 2 i .
Then, we need to estimate the asymptotics of ˜ E ( Y n [ X 1 n ] s 1 · · · [ X kn ] s k ) and ˜ E Y 2 n , where the calculation of ﬁrst moment is somewhat similar to calculations in the proof of Theorem 2.1.
The following lemma gives an estimation for the second moment of Y n .
Lemma 3.4.
[Equation]
After verifying all the conditions of Theorem 3.3 (see the Appendix) the ﬁrst half of Theorem 1. follows directly.
Conclusion
Community detection is of great interest in many ﬁelds such as network analysis, statistics and computer vision.
Hypergraphs have been widely used to model higher-order relationships among data.
The SBM (graph-based) model is popular in community detection but only directly models pair-wise interactions.
Its hypergraph version, HSBM, likewise serves as a benchmark for clustering algorithms on higher order interaction (hypergraph) data.
This paper provides theoretical analyses on the community detection thresholds in the sparse HSBM.
Speciﬁcally, we address a previous conjecture that there exists a sharp threshold on the model parameters for community detection to be possible.
We proved that above the threshold, it is possible to asymptotically almost surely construct a bi-partition of the hypergraph positively correlated with the true bi-partition.
Moreover, below the threshold, we show that community detection is informationtheoretically impossible.
Importantly, we also provide consistent estimators for the model parameters, when the detection is possible.
Our estimators require the counting of the number of loose cycles in the hypergraph.
To our best knowledge, there is no efﬁcient algorithm to count the number of loose cycles in hypergraphs in general.
Therefore, from the algorithmic side, it might be more practical to count the number of non-backtracking/self-avoiding walks of the same length [26, 33], which can be computed by matrix multiplication efﬁciently.
Nevertheless, from the theoretical side, conﬁrming the positive side of the threshold conjecture by analysing the number of loose cycles has not been done before.
Moreover, providing more practical estimators (or examining whether one can estimate the number of loose cycles sufﬁciently accurate to still use the estimators of model parameters we provide), is a worthy topic of future research.
Note that the non-uniform hypergraphs can be viewed as a ‘superposition’ of a collection of uniform hypergraphs.
Thus it may be possible to generalise our proofs to cover non-uniform hypergraphs in future work.
Considering other variants of the hypergraph community detection problem, for instances, multi-community structures, extremely sparse scenario ( p n , q n ⌧ 1 /n d  1 ) , moderate sparse scenario ( p n , q n = O (log( n ) /n
d  1 )) , dense scenario ( p n , q n  1 /n d  1 ) , are future research directions that would complete the work presented here.
Acknowledgments and Disclosure of Funding
The authors would like to express their gratitude to three anonymous referees for their helpful suggestions and comments.
{+This work was partly supported by the Australian Research Council grants DP
+}
(D. Suter & E. Zhang).
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section 2.
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] See the abstract and introduction. (b) Did you describe the limitations of your work?
[Yes] See the last two paragraphs of the conclusion section.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] We have read the ethics review guidelines.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See the main results subsection.
(b) Did you include complete proofs of all theoretical results?
[Yes] Some proofs are added in the Appendix.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]