Estimation of Number of Communities in Assortative Sparse Networks
Authors: Neil Hwang, Jiarui Xu, Shirshendu Chatterjee, Sharmodeep Bhattacharyya
Keywords: networks, number of communities, Bethe-Hessian, sparse networks, stochastic block model
Abstract: Most community detection algorithms assume the number of communities, $K$, to be known \textit{a priori}. Among various approaches that have been proposed to estimate $K$, the non-parametric methods based on the spectral properties of the Bethe Hessian matrices have garnered much popularity for their simplicity, computational efficiency, and robust performance irrespective of the sparsity of the input data. Recently, one such method has been shown to estimate $K$ consistently if the input network is generated from the (semi-dense) stochastic block model, when the average of the expected degrees ($\tilde{d}$) of all the nodes in the network satisfies $\tilde{d} \gg \log(N)$  ($N$ being the number of nodes in the network). In this paper, we prove some finite sample results that hold for $\tilde{d} = o(\log(N))$, which in turn show that the estimation of $K$ based on the spectra of the Bethe Hessian matrices is consistent not only for the semi-dense regime, but also for the sub-logarithmic sparse regime when $1 \ll \tilde{d} \ll \log(N)$. Thus, our estimation procedure is a robust method for a wide range of problem settings, regardless of the sparsity of the network input.