H YPERGRAPH C ONVOLUTIONAL N ETWORKS VIA E QUIVALENCY BETWEEN H YPERGRAPHS AND U NDIRECTED G RAPHS
Anonymous authors
Paper under double-blind review
A BSTRACT
As a powerful tool for modeling the complex relationships, hypergraphs are gaining popularity from the graph learning community.
However, commonly used algorithms in deep hypergraph learning were not speciﬁcally designed for hypergraphs with edge-dependent vertex weights (EDVWs).
To ﬁll this gap, we build the equivalency condition between EDVW-hypergraphs and undirected simple graphs, which enables utilizing existing undirected graph neural networks as subroutines to learn high-order interactions induced by EDVWs of hypergraphs.
Speciﬁcally, we deﬁne a generalized hypergraph with vertex weights by proposing a uniﬁed random walk framework, under which we present the equivalency condition between generalized hypergraphs and undigraphs.
Guided by the equivalency results, we propose a Generalized Hypergraph Convolutional Network (GHCN) architecture for deep hypergraph learning.
Furthermore, to improve the longrange interactions and alleviate the over-smoothing issue, we further propose the
Simple Hypergraph Spectral Convolution (SHSC) model by constructing the Discounted Markov Diffusion Kernel from our random walk framework.
Extensive experiments from various domains including social network analysis, visual objective classiﬁcation, and protein fold classiﬁcation demonstrate that the proposed approaches outperform state-of-the-art spectral methods with a large margin.
I NTRODUCTION
Hypergraphs, whose edges link to the arbitrary number of vertices to model high-order relationships, attract much attention recently from researchers in the graph learning community (Feng et al., 2019; Jiang et al., 2019; Zhang et al., 2021).
The topology of hypergraph can be considered to be embedded in vertex weights, which represents the connection strength between hyperedges and vertices (Chitra & Raphael, 2019).
The vertex weights can be divided into two categories, edgeindependent and edge-dependent (Chitra & Raphael, 2019; Hayashi et al., 2020), depending on whether the vertex weights are related to the incident hyperedges or not (EDVW-hypergraph implies that each vertex v is assigned a weight q e ( v ) ∈ R for each incident hyperedge e ).
Many hypergraph neural networks have been proposed (Feng et al., 2019; Jiang et al., 2019; Yadati, 2020; Huang & Yang, 2021) to handle hypergraphs with edge-independent vertex weights (EIVW-hypergraph).
However, rigorous study for hypergraphs with edge-dependent vertex weights (EDVW-hypergraph) is still lacked, though they enjoy stronger expressive power (Ding & Yilmaz, 2010;
Huang et al., 2010; Li et al., 2018a; Zhang et al., 2018).
In this work, we are devoted to spectral-based methods, which are considered to be robust and to allow for simple model property analysis (Wu et al., 2020).
Unfortunately, the spectral theory of
EDVW-hypergraph has not been well established.
A recent work (Chitra & Raphael, 2019) provides the spectral theory for EDVW-hypergraphs, and indicates that the deﬁned EDVW-hypergraphs is equivalent to digraphs.
Though digraph convolutional networks can be directly used for EDVWhypergraphs, most existing convolutional algorithms for digraphs are more or less related to undirected graphs, e.g., (Monti et al., 2018; Li et al., 2020;
Ma et al., 2019; Tong et al., 2020).
Essentially, they transform digraphs to undigraphs with various techniques due to the challenges of directed graph Laplacian.
In addition, key theoretical analysis of digraph convolutional networks properties,
[Figure]
such as expressive power (Xu et al., 2019) or the performance of models in deep layers (Li et al., 2018b),
yet remains open.
An arguably more succinct way for EDVW-hypergraph learning is to use undigraph convolutional networks.
Compared with digraphs, the undigraph convolutional networks have been extensively studied and there exist various effective strategies for analyzing their properties, to name a few (Kipf & Welling, 2017; Chen et al., 2020; Zhu & Koniusz, 2021).
Furthermore, one can take established techniques on undigraphs as a subroutine for hypergraph learning, which would largely ease hypergraph learning in real-world scenarios.
It remains unclear, however, whether EDVW-hypergraphs can be made equivalent to undigraphs in a principled way, thereby hindering this promising prospect.
In this paper, we focus on constructing the equivalency condition between hypergraphs and undigraphs.
This makes it possible to utilize undigraphs as low-order encoders of hypergraphs, and thus enabling hypergraph learning directly from the equivalent undirected graphs by means of undigraph convolutional networks.
Along this route, we have made the following contributions:
1) We deﬁne a generalized hypergraph capturing EDVWs via designing a two-step uniﬁed random walk framework.
In addition, we construct the equivalency conditions to ﬁll the gap between the generalized hypergraph and corresponding undigraph.
2) We present a uniﬁed Laplacian from the equivalency conditions and propose two spectral convolution models (GHCN, SHSC) for EDVW-hypergraphs learning.
We analyze the over-smoothing issues (Li et al., 2018b) based on their spectral properties deduced from equivalency conditions.
3) Extensive experiments across various domains (social network analysis, visual objective classiﬁcation and protein modeling) demonstrate the effectiveness of the proposed methods (GHCN and
SHSC) for both EIVW-hypergraph and EDVW-hypergraph learning tasks.
Notably, we are ﬁrst to adopt EDVW-hypergraphs for protein structures and obtain signiﬁcant performance boost.
Due to space limit, more discussions on related works are deferred to Appendix A.
T HE T HEORY OF E QUIVALENCY
Notations.
I ∈ R n × n denotes the identity matrix, and 1 ∈ R n represents the vector with ones in each component.
We use boldface letter x ∈ R n to indicate an n -dimensional vector, where x ( i ) is the i th entry of x .
We use a boldface capital letter A ∈ R m × n to denote an m by n matrix and use A ( i, j ) to denote its ij th entry.
Let G ( V , E , ω ) be a graph with vertex set V , edge set E , and edge weights ω .
Let H ( V , E , W , Q ) be the hypergraph with vertex set V , edge set E .
Let w ( e ) denote the weight of hyperedge e .
W ∈ R |E|×|E| is the edge-weights diagonal matrix with entries
W ( e, e ) = w ( e ) .
Q denotes edge-vertex-weights matrix with entries Q ( u, e ) =
q e ( u ) ∈ R if u ∈ e and 0 if u / ∈ e .
Q is said to be edge-independent if q e ( u ) =
q ( u ) ∈ R for all e (cid:51) u , and called edge-dependent otherwise (Chitra & Raphael, 2019).
If q e ( u ) equals to 1 for all linked u and e , Q would reduce to the binary incident matrix H , which is widely used to represent the structure of hypergraph (Zhou et al., 2006; Carletti et al., 2021).
The matrix Q can also be viewed as a weighted incident matrix.
Note that a graph is a special case of a hypergraph with hyperedge degree | e | = 2 .
We say a hypergraph is connected when its unweighted clique graph (Chitra & Raphael, 2019) is connected and we assume hypergraphs are connected.
Throughout this paper, equivalency refers to equivalency between hypergraphs and undigraphs if not otherwise speciﬁed.
2. U NIFIED R ANDOM W ALK AND G ENERALIZED H YPERGRAPHS
Traditionally, hypergraph random walk is deﬁned by a two-step manner (Zhou et al., 2006; Ducournau & Bretto, 2014) (Fig. 2).
Then, Chitra & Raphael (2019) raise a new random walk involving the edge-dependent vertex weights into the second step and build the equivalency between EDVWhypergraphs and digraphs.
However, due to the limitation of their hypergraph deﬁnition, they fail to answer whether the EDVW-hypergraphs can be equal to undigraphs.
So in this part, we integrate the existing two-step random walk methods (Chitra & Raphael, 2019; Carletti et al., 2020; 2021) to obtain a comprehensive uniﬁed random walk framework with the vertex weights added into the ﬁrst step, based on which we further deﬁne a generalized hypergraph.
Deﬁnition 1 (Uniﬁed Random Walk on Hypergraphs)
The uniﬁed random walk on a hypergraph
H is deﬁned in a two-step manner:
Given the current vertex u ,
Step I : choose an arbitrary hyperedge e incident to u , with the probability
[Equation]
Step II : choose an arbitrary vertex v from e , with the probability
[Equation]
where w ( e ) denotes the hyperedge weight; Q 1 ( u, e ) denotes the contribution of vertex u to hyperedge e in the ﬁrst step while Q 2 ( v, e ) represents the contribution of vertex v to hyperedge e in the second step; and ρ ( · ) is a real-valued function, for example, a power function ( · ) σ .
Suppose δ ( e ) :
=
(cid:80) v ∈V Q 2 ( v, e ) is the degree of hyperedge, and d ( v ) :
= (cid:80) e ∈E w ( e ) δ ( e ) ρ ( δ ( e ))
Q 1 ( v, e ) is the degree of vertex v .
The transition probability of our uniﬁed random walk on a hypergraph from vertex u to vertex v is:
[Equation]
Here, P ( u, v ) can be written in a |V| × |V| matrix form: P = D − 1 v Q 1 W ρ ( D e ) Q (cid:62) 2 , where D v ,
D e are diagonal matrices with entries D v ( v, v ) =
d ( v ) and D e ( e, e ) =
δ ( e ) , respectively.
ρ ( D e ) represents the function ρ acting on each element of D e .
Notably, the ρ does not always work properly.
We present the failure conditions in the Appendix G.4.
More detailed intuition of our purpose to design P ( u, v ) can be found in Appendix B.2.
Under our framework, the existing hypergraph random walk (Zhou et al., 2006; Chitra & Raphael, 2019; Carletti et al., 2020; 2021) can be seen as special cases with speciﬁc p 1 , p 2 (see Appendix B.3).
The deﬁnition of our framework is lazy since it allows self-loops ( P ( v, v ) > 0 ).
The non-lazy version is provided in Appendix C. Based on the framework, we deﬁne the generalized hypergraph for constructing equivalency between EDVW-hypergraphs and undigraphs as follows.
Deﬁnition 2 (Generalized hypergraph)
A generalized hypergraph is a hypergraph associated with the uniﬁed random walk in Deﬁnition 1, denoted as H ( V , E , W , Q 1 , Q 2 ) .
Here, Q 1 , Q 2 are vertex weights matrices, each of which can be edge-independent or edge-dependent.
2. T HE E QUIVALENCY B ETWEEN H YPERGRAPHS AND UNDIGRAPHS
In this part, we provide the condition under which the generalized hypergraph is equivalent to weighted undirected clique graph.
By establishing the equivalency, we can view graphs as loworder encoders of EDVW-hypergraphs, so any undigraph technique can be used as a subroutine for hypergraph learning.
The clique graph of H ( V , E , W , Q 1 , Q 2 ) is denoted as G C , which is a unweighted graph with vertices V and edge set { ( u, v ) : u, v ∈ e, e ∈ E} .
Actually, G C turns all hyperedges into cliques.
We ﬁrst present the deﬁnition of the equivalency in connection with Markov chains.
Deﬁnition 3 (Chitra & Raphael (2019))
Let M 1 , M 2 be the Markov chains with the same ﬁnite state space, and let P M 1 and P M 2 be their probability transition matrices, respectively.
M 1 , Mare equivalent if P M 1 u,v = P M 2 u,v for all states u and v .
[Figure]
The equivalency between digraph and hypergraph is straightforward and we defer to Appendix E.
On the other hand, it is easy to understand that any simple undigraph can be constructed as a special generalized hypergraph, while the converse is not necessarily true.
We provide a counterexample in Appendix G.4.
This result implies that undigraph convolutional networks can not trivially be extended to generalized hypergraphs.
So, we give one of the implicit equations for formulating the equivalency problem to explore the explicit equivalency conditions.
Lemma 1 Let H ( V , E , W , Q 1 , Q 2 ) denote the generalized hypergraph in Deﬁnition 2.
Let F ( Q 1 ,Q 2 ) ( u, v ) :
= (cid:80) e ∈E w ( e ) ρ ( δ ( e ))
Q 1 ( u, e ) Q 2 ( v, e ) and T ( Q ) ( u ) :
= (cid:80) e ∈E w ( e ) δ ( e ) ρ ( δ ( e ))
Q ( u, e ) .
When Q 1 , Q 2 satisﬁes the following equation
[Equation]
there exists a weighted undirected clique graph G C such that a random walk on H is equivalent to a random walk on G C with edge weights ω ( u
[Equation]
=
[Equation]
= 0 and 0 otherwise.
The proof can be found in Appendix G.2.
Next, we derive the explicit condition under which random walks on hypergraphs are equivalent to undigraphs.
Theorem 1 (Equivalency between generalized hypergraph and weighted undigraph) Let
H ( V , E , W , Q 1 , Q 2 ) denote the generalized hypergraph in Deﬁnition 2.
When H satisﬁes any of the condition bellow:
Condition (1) Q 1 and Q 2 are both edge-independent;
[Equation]
there exists a weighted undirected clique graph G C such that a random walk on H is equivalent to a random walk on G C .
It is easy to verify that conditions (1) and (2) satisfy equation 4 (proof in Appendix G.3).
This theorem brings insights from three aspects: i) Hypergraph foundation.
The equivalency itself is a fundamental problem and remains open before this work (Agarwal et al., 2006; Chitra & Raphael, 2019).
Our conclusion provides more adaptive and explores more essential conditions thanks to the unity of our random walk and the generalization of hypergraph deﬁned (Figure 3).
Notably, Chitra & Raphael (2019) shows that the equivalency condition is: Q 1 = H and Q 2 is edge-independent, which can be viewed as a special case of the condition (1), while Theorem 1 implies the equivalency is not only related to the dependent relationships between vertex weights and edges, but also to whether the vertex weights used in the ﬁrst and second step of the uniﬁed random walk are proportional.
Furthermore, thanks to the introduced Q 1 in the generalized hypergraph, we can obtain the equivalency between EDVW-hypergraph and undigraph (i.e. condition (2) ), ﬁlling an important gap in the equivalency theory of EDVW-hypergraph.
ii)
Providing a theoretical basis for hypergraph applications.
The condition (2), containing both the edge-independent vertex weights and edge-dependent vertex weights cases which match different hypergraph applications (Ding & Yilmaz, 2010; Zeng et al., 2016; Zhang et al., 2018), provides those with theoretical explanations.
The condition (2) also reveals that the existing random walks (Carletti et al., 2020; 2021) on hypergraphs are equivalent to undigraph.
iii)
Hypergraph learning.
The equivalent undigraphs can be viewed as the lower-order encoders of hypergraphs.
Thus, one can obtain hypergraph representations directly by exploiting the undigraph learning methods.
Especially, the condition (2), which implies that
an
EDVW-hypergraph can be equivalent to an undigraph, gives the theoretical guarantee for learning hypergraphs without losing edge-dependent vertex weights via undigraph neural networks.
Next, we provide the spectral theory for the generalized hypergraphs.
2. G ENERALIZED H YPERGRAPH L APLACIAN AND I TS S PECTRAL P ROPERTIES
To build a bridge between spectral convolution and equivalency, we derive the equivalent undirected weighted graph Laplacian based on Lemma 1 and Thm 1.
Then the graph Laplacian can be considered as the Laplacian of generalized hypergraph.
Starting from the deduced Laplacian, it is direct and convenient to construct the spectral convolution for hypergraphs.
Formally, we have
Corollary 1 Let H ( V , E , W , Q 1 , Q 2 ) be the generalized hypergraph in Deﬁnition 2.
Let ˆ D v be a |V| × |V| diagonal matrix with entries
ˆ D v ( v, v ) :
= ˆ d ( v ) :
= (cid:80) e ∈E w ( e ) δ ( e ) ρ ( δ ( e ))
Q 2 ( v, e ) .
No matter H satisﬁes condition (1) or condition (2) in Thm.
1, it obtains the uniﬁed explicit form of stationary distribution π
and Laplacian matrix L as:
[Equation]
There are two observations from Corollary 1: (i)
This stationary distribution is different from the classical π ( v ) =
d ( v ) / (cid:80) v d ( v ) (Zhou et al., 2006), which depends on the vertex degree d ( v ) .
Our π ( v ) is related to ˆ d ( v ) , which means that we cannot trivially extend from classical theory.
The detailed proof is deferred to Appendix G.6; (ii) Both π and L are only related to Q 2 , independent of
Q 1 , implying possible information loss of Q 1 under the setting required an asymmetrical two-steps random walk(i.e.
Q 1 (cid:54) =
Q 2 ) to construct a hypergraph.
Meanwhile, L can be viewed as a normalization Laplacian led by the equivalent undigraph G C with adjacency matrix Q 2 W ρ ( D e ) Q (cid:62) 2 , based on which we can design the EDVW-hypergraph spectral convolutions.
Corollary 1 implies that no matter Q 1 , Q 2 are both edge-independent or Q
= Q 2 , we can directly use the Laplacian matrix L to analyze the spectral properties.
The spectral properties of Laplacian (Chung & Graham, 1997) are vital in the research of graph neural networks to design a stable and effective convolution operator.
Therefore, we deduce two important conclusions concerning eigenvalues of L as the basic theory of Laplacian application under the equivalency conditions in
Thm. 1.
One claims the eigenvalues range of L is [0 , 2] (details in Appendix G.7), and the other describes the rate of convergence of our uniﬁed random walk listed bellow (see Appendix G.8)
Corollary 2 Let H ( V , E , W , Q 1 , Q 2 ) be the generalized hypergraph in Deﬁnition 1.
When H satisﬁes any of two conditions in Thm. 1, let L and π be the hypergraph Laplacian matrix and stationary distribution from Corollary 1.
Let λ H denote the smallest nonzero eigenvalue of L .
Assume an initial distribution f with f ( i ) =
1 ( f ( j ) = 0 , ∀ j (cid:54) = i ) which means the corresponding walk starts from vertex v i .
Let p ( k ) =
fP k be the probability distribution after k steps uniﬁed random walk where P denotes the transition matrix, then p ( k ) ( j ) denotes the probability of ﬁnding the walker in vertex v j after k steps.
We have:
[Equation]
For hypergraph learning, Corollary 2 implies that the spectral convolution deduced from the Laplacian in Corollary 1 might suffer from the over-smoothing issue (Zhu & Koniusz, 2021).
Appendix
F.4 gives a theoretical perspective for analyzing the over-smoothing issue based on this Corollary, which further inspires the design of convolutions in Section 3 to alleviate the issue.
T WO S PECTRAL C ONVOLUTIONS FOR H YPERGRAPHS
Intuitively, there are two possible routes for designing EDVW-hypergraph neural networks: 1.
message passing; 2.
spectral-based methods.
Message passing has proved to be a powerful tool for extracting information from hypergraphs or graphs.
However, on the one hand, researchers usually adopt heuristic ideas to design message-passing models, leading to a lack of theoretical guarantees (Huang & Yang, 2021).
This makes it difﬁcult to analyze properties of corresponding neural networks directly, such as over-smoothing issue (Li et al., 2018b).
On the other hand, spectral-based methods not only have a solid foundation in graph signal processing (Kipf & Welling, 2017; Feng et al., 2019), but also effectively inspires the design of message passing techniques (Xu et al., 2019).
Here, we design two hypergraph spectral convolutions for hypergraph learning based on equivalency conditions in Thm. 1.
As the Laplacian L enjoys the same formula under any of the two equivalency conditions, algorithms deduce by L would be adaptive to any generalized hypergraph satisﬁed
Thm. 1, including EIVW-hypergraph and EDVW-hypergraph.
3. G ENERALIZED S PECTRAL H YPERGRAPH C ONVOLUTION
We denote K as Q 2 W ρ ( D e ) Q (cid:62) 2 for expression simplicity.
Then the uniﬁed hypergraph Laplacian matrix L in Corollary 1 can be expressed as L =
I − ˆ D − 1 / 2 v
[Equation]
Similar to Defferrard et al.
, we approximate the convolutional kernel with Chebyshev polynomial to avoid eigenvalue decomposition of L .
Finally, the following Generalized Hypergraph Convolutional Network ( GHCN ) is obtained by introducing the re-normalization technique (Kipf & Welling, 2017):
[Equation]
where ˜ T :=
˜ D − 1 / 2 v ˜ K ˜ D − 1 / 2 v , and ˜ K
= K
+ I can be regarded as the weighted adjacency matrix of the equivalent undigraph with self-loops, and
˜ D ( v, v ) =
(cid:80) u ∈V ˜ K ( v, u ) .
Θ is a learnable parameter matrix and ψ ( · ) denotes an activation function.
More details of derivation can be found in Appendix
F.1.
Most of existing hypergraph Laplacians (Zhou et al., 2006; Carletti et al., 2020; 2021) can be viewed as special forms of the Laplacian matrix L and can be derived the special convolutions of
GHCN.
The biggest advantage of GHCN is that it can handle both EIVW-hypergraphs and EDVWhypergrpahs.
Besides, the unity of random walk makes it possible to aggregate more ﬁne-grained information.
Note that HGNN (Feng et al., 2019) can be viewed as a speciﬁc case of GHCN without the re-normalization trick ( ρ ( · ) = ( · ) − 1 , Q = H ).
However, Corollary 2 indicates that initial signals or features intend to converge to a certain vector, causing vertices hard to distinguish between each other.
This implies that GHCN also suffers from over-smoothing issue (Li et al., 2018b), which limits the depth of the convolution model and hinders the aggregation of long-range information.
In fact, the analysis also suggests that HGNN (Feng et al., 2019) suffers from the over-smoothing issue.
Next, we would propose a new spectral convolution to alleviate this issue.
3. A S IMPLE S PECTRAL C ONVOLUTION B ASED ON M ARKOV D IFFUSION K ERNEL
In order to aggregate long-range information, we propose the discounted Markov Diffusion process to aggregate the information of vertices and propose a new convolutional architecture for hypergraphs.
Discounted Markov Diffusion Kernel for Hypergraphs.
A diffusion process on a generalized hypergraph can be understood as a process of information extracting among vertices.
To generate the underlying feature map ( Z ( t ) ), we ﬁrst utilize the transition probabilities of the Markov chain in
Deﬁnition 1 to design a discounted average visiting rate : ¯ v ik ( t ) = 1 t (cid:80) tτ
=
α τ Pr( s ( τ ) =
k | s (0) = i ) , where t ∈ Z
++ .
Notably, we introduce a discount factor α ∈ (0 , 1] to exponentially weaken the long-step transition probabilities from vertex i to vertex k .
Specially, we set ¯ v ik (0)
=
if and only if i = k , otherwise ¯ v ik (0) = 0 .
Assume that ¯ v i ( t ) =
(¯ v i 1 ( t ) , · · · , ¯ v i |V| ( t )) (cid:62) .
Then, we adopt the diffusion distance d ij ( t ) =
(cid:107) ¯ v i ( t ) − ¯ v j ( t ) (cid:107)
=
(cid:13)(cid:13)
Z (cid:62) ( t )(¯ v i (0) − ¯ v j (0))
(cid:13)(cid:13) 2 2 proposed in Pons &
Latapy (2005) to generate our discounted Markov diffusion kernel K MD ( t ) =
Z ( t ) Z (cid:62) ( t ) , where
Z ( t ) =
(cid:80) tτ
=
α τ t P τ can be viewed as the weighted sum of the transition matrix (derivation in
Appendix F.2).
The intuition behind this diffusion distance is that if two vertices diffuse similarly based on hypergraph random walk, we are supposed to take them similar.
Then, the inﬂuence of the two vertices to other vertices in the hypergraph is considered in a similar manner.
Simple Hypergraph Spectral Convolution (SHSC).
Inspired by the diffusion kernel-based GCNNs from Zhu & Koniusz (2021), based on our discounted Markov diffusion kernel, we propose a
Simple Hypergraph Spectral Convolution ( SHSC ) as follows:
[Equation]
Recall ˜ T is the transition matrix we used in equation 7, ψ ( · ) is the activation function to map the node representation of output, and Θ is the parameter of ﬁlter to be learned during training.
SHSC is able to balance the global information aggregation and the vertex’s own information by introducing a hyperparameter β ∈ [0 , 1] to control the self-loops (˜ T )
= I .
Through the discounted Markov
Diffusion process, our SHSC could gain stronger local and weaker global information, thereby improving the expressive power in the deep layers.
On the other hand, SHSC uses only one linear layer, which greatly improves the computational efﬁciency and effectively alleviates the overﬁtting problem.
Actually, SHSC is a spatial-based model while we can analyze it from a spectral-based perspective, as SHSC can be viewed as a polynomial ﬁlter with speciﬁc coefﬁcients (see Appendix
F.3).
Finally, we study why SHSC can relieve over-smoothing.
Speciﬁcally, we utilize Corollaryto generate a lower-bounded quantity of the deﬁned over-smoothing energy , based on which we show that SHSC can relieve the over-smoothing issue compared to multi-layer GHCNs (details in
Appendix F.4).
[Table]
E MPIRICAL S TUDIES
[Table]
In this section, we evaluate our proposed methods on four tasks: citation network classiﬁcation, visual object classiﬁcation, protein quality assessment (regression), and fold classiﬁcation.
For simplicity, we set ρ ( · ) to be a power function ( · ) σ ( σ is a hyper-parameter) and additional experiments to compare the performance with different ρ is deferred to Appendix I.10.
The weight matrix of edges, we set, to be an identity matrix by default in our GHCN and SHSC models.
Notably, although GHCN and SHSC are designed for
EDVW-hypergraph learning, they work for EIVWhypergraph as well thanks to the uniﬁed Laplacian in Corollary 1.
In our experiments, citation network classiﬁcation belongs to EIVW-hypergraph learning tasks, and visual object classiﬁcation and protein learning are EDVW-hypergraph learning tasks.
For baselines designed for EIVW-hypergraph, we replace Q 1 , Q 2 with H .
Due to space limitations, we defer additional experimental results on Over-smoothing Analysis,
Hyper-parameter Sensitivity Analysis, and Ablation Analysis to Appendix I.6, I.7, I.9, respectively.
The insights are: 1) SHSC can efﬁciently alleviate over-smoothing; 2)
The hyper-parameters σ, β, α are sensitive at a relatively large range but insensitive in a small range of the performance optimum; 3) The re-normalization trick and edge-dependent vertex weights are both effective.
In addition, we provide the computational complexity analysis for our models in Appendix I.11.
4. C ITATION N ETWORK C LASSIFICATION
This is a semi-supervised node classiﬁcation task.
The datasets include co-authorship and co-citation datasets: PubMed, Citeseer, Cora (Sen et al., 2008), and DBLP (Rossi & Ahmed, 2015).
We adopt the same public datasets ( https://github.com/malllabiisc/HyperGCN ) and train-test splits of Yadati et al.
Note these datasets satisfy the condition that Q 1 = Q 2 = H .
So this experiment can be regarded as a special case of the applications of our models.
For baselines, we include Multi-layer perceptron(MLP), HNHN (Dong et al., 2020), HGAT (Ding et al., 2020),
UniGNN (Huang & Yang, 2021), and two recent spectral-based hypergraph convolutional neural networks (HGCNNs): HGNN (Feng et al., 2019) and HyperGCN (Yadati et al., 2019). Each [Table]
[Table]
Comparison with SOTAs.
As shown in Table 19, the results successfully verify the effectiveness of our models and achieve a new SOTA performance across all ﬁve datasets.
Notably, both GHCN and SHSC have an average performance improvement of 4.88% and 5.83% over the previous spectral-based methods, respectively.
On the other hand, SHSC achieves the best result at a deep layer and gains better performance than GHCN on most datasets, which demonstrates the beneﬁts of deep model and the long-range information in hypergraphs.
It’s worth noting that our methods gain superior performance on disconnected datasets compared to HGNN.
HGNN shows poor performance on disconnected datasets, mainly due to the row in the adjacency matrix of equivalent undigraph corresponding to an isolated point is 0, resulting direct loss of its vertex information (an example can see Fig.
4 in appendix).
And our methods utilize the renormalization trick, which can maintain the features of isolated vertices during aggregation.
Another important observation is that HyperGCN has a high standard deviation, revealing its poor generalization.
Conversely, SHSC has lower bias and standard deviation than others, showing better generalization.
4. V ISUAL O BJECT C LASSIFICATION
This experiment is about semi-supervised learning.
We employ two public benchmarks: Princeton ModelNet40 dataset (Wu et al., 2015) and the National Taiwan University (NTU) 3D model dataset (Chen et al., 2003) to evaluate our methods.
We follow HGNN (Feng et al., 2019) to preprocess the data by MVCNN (Su et al., 2015) and GVCNN (Feng et al., 2018).
Finally, we use the datasets provided by its public Code ( https://github.com/iMoonLab/HGNN ).
More details can be found in Appendix I.4.
Results.
Table 2 depicts that our methods signiﬁcantly outperform the image-input or point-input methods.
These results demonstrate that our methods can capture the similarity of objects in the feature space to improve the performance of the classiﬁcation task.
Table 20 compares our methods with HGNN and uniGNN on NTU and ModelNet40.
From the results, we can see that our methods outperform HGNN on both single modality and multi-modality (BOTH) datasets and SHSC achieves much better performance on multi-modality compared with others.
These results reveal that our
SHSC has the advantage of combining such multi-modal information through concatenating the weighted incidence matrices ( Q ) of hypergraphs, which means merging the multi-level hyperedges.
4. P ROTEIN Q UALITY A SSESSMENT (QA) AND F OLD C LASSIFICATION
Protein QA and fold classiﬁcation are vital for mining the property of proteins.
However, to the best of our knowledge, most researchers represented a protein as a sequence or a simple graph (Baldassarre et al., 2020; Hermosilla et al., 2021) in the deep learning area.
So we’d like to investigate whether hypergraphs are better than simple graphs for protein modeling, and the following results conﬁrm this conjecture.
Protein hypergraphs.
A protein is a chain of amino acids (residues) that will fold to a 3D structure.
To simultaneously model protein sequence and spatial structure information, we build sequence hyperedges and distance hyperedges: we choose τ consecutive amino acids ( v i , v i +1 , · · · , v i + τ ) to form a sequence hyperedge, and choose amino acids whose spatial Euclidean distance is less than the threshold (cid:15) > 0 to form a spatial hyperedge, where v i ( i = 1 , · · · , | S | ) represents the i -th amino acids in the sequence.
Appendix I.5 contains more details.
Protein Quality Assessment (QA).
Protein QA is used to estimate the quality of computational protein models in terms of divergence from their native structure.
It is a regression task to predict how close the decoy is to the unknown, native structure.
[Table]
Inspired by Baldassarre et al.
(2020), we train our models on Global Distance Test
Score (Zemla, 2003), which is the globallevel score, and the Local Distance Difference Test (Mariani et al., 2013), an aminoacids-level score.
The loss function of QA is deﬁned as the Mean Squared Error (MSE):
L g =
MSE ( P gpred − GDT TS) , L l = (cid:80) | S | i =1 MSE ( P lpred i − LDDT i ) where P gpred and P lpred denote predicted global and local score, respectively.
We jointly learn node and graph embeddings and the losses are weighted as L total = µ L l + (1 − µ ) L g , where µ is a hyper-parameter.
We use the data from past years’ editions of CASP, including CASP10-13.
CASP10,11,13 are used for training and validation, and CASP12 is used for testing.
Results.
Table 4 shows that our methods outperform most of SOTAs, which demonstrates our methods can learn protein more efﬁciently.
The outstanding performance of SHSC indicates that it is able to jointly learn better the node and graph level representation.
But SHSC trained to predict only global scores obtains R = 0.712, which suggests that the local information can help the assessment of the global quality.
[Table]
Protein Fold Classiﬁcation.
This task is critical for studying the relationship between protein structure and function, and protein evolution.
We use the well-known SCOPe 1.75 dataset (Hou et al., 2018) and the cross-entropy loss.
Results.
Table 5 depicts that our proposed GHCN outperforms all others.
It is manifest in the table that HGCNN-based methods GHCN and HGNN achieve much better performance compared to GCNN-based.
However, it is worth noting that our proposed SHSC, does not seem to offer signiﬁcant advantages over GHCN and HGNN.
This can be understood in the following lens:
SGHC is used to alleviate the over-smoothing problem, which has a large impact on the node-level task, so our SHSC is more suitable for node-level tasks, while for the pure graph-level tasks, SHSC is too simple to offer good performance.
D ISCUSSIONS AND F UTURE W ORK .
Despite the good results, there are some limitations which worth further explorations in the future: 1) We just provide sufﬁcient conditions for the equivalency to undigraphs, but the necessary condition is still unclear.
With the equivalency between hypergraphs and undigraphs, we have extended the undigraph-based GCNNs to hypergraph learning in this paper.
Meanwhile, the equivalency between hypergraphs and digraphs would allow one to extend digraph-based GCNNs to potentially learn richer information in hypergraphs.
3) One could use the proposed uniﬁed random walk on hypergraphs to devise new clustering algorithms for hypergraph partitioning.
We make every effort to include a thorough experimental
E THICS S TATEMENT AND B ROADER I MPACT
In the work, we proposed a uniﬁed random walk framework on hypergraphs, based on which we study the equivalence between hypergraphs and graphs, and show that the uniﬁed random walk on
EDVW-hypergraphs can not be equivalent to undigraphs without some speciﬁc conditions.
Meanwhile, we lead to the Laplacian of our framework and analyze the spectral properties of it, especially focusing on the condition equivalent to an undigraph.
Furthermore, we develop a Generalized Hypergrpah Convolutional Network and a Simple Hypergraph Spectral Convolution for hypergraph learning, based on the condition equivalent to undigraph.
Our research of the equivalency conditions between hypergraph and graph opens up many more possibilities in designing more comprehensive graph-based methods to solve the hypergraph problem.
The experimental tasks used in our work including social networks and academic networks analysis, visual object classiﬁcation, and protein learning, suggest that GHCN and SHSC can be applied for beneﬁcial purposes, such as Rumour Detection, Crime Identiﬁcation, Protein Function Prediction, etc.
Meanwhile, we have to be aware of possible negative impacts, such as using our model for telecom fraud on social networks, placing a high degree of trust in our model and obtain an incorrect interpretation of the results, etc.
In addition, we should be vigilant about the potential unemployment issue due to the reduced amount of the need of labeling by human beings.
R EPRODUCIBILITY S TATEMENT
All the datasets are publicly available as described in the main text.
Our efforts to ensure reproducibility include the following aspects: 1) A sampled code is provided in the supplementary material.
The proofs and derivations are provided in Appendix G, F. 3) More details of experimental conﬁgurations are provided in Appendix I.
Appendix C ONTENTS A Related Work
A.1 Simpliﬁed Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.2 Detailed Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B Relations with Previous Random Walks on Hypergraph
B.1 A Classical Random walk on Hypergraph . . . . . . . . . . . . . . . . . . . . . .
B.2 Intuition and Analysis of The Uniﬁed Random Walk on Hypergraph . . . . . . . .
B.3 Special Cases of The Uniﬁed Random Walk Framework and Laplacian . . . . . . .
C Uniﬁed Non-lazy Random Walks on Hypergraph
D Typical Hypergraph Convolution (HGNN) and Its Spectral Analysis
D.1 Typical Spectral Hypergraph Convolution . . . . . . . . . . . . . . . . . . . . . .
D.2 Spectral Analysis of HGNN . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E The Relationship between Generalized Hypergraphs and Digraphs
F Derivation and Analysis of GHCN and SHSC
F.
Derivation of GHCN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F.
Derivation of The Discounted Markov Diffusion Kernel . . . . . . . . . . . . . . .
F.
Derivation of SHSC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F.
Over-smoothing Analysis of GHCN
G Details of Theories and Proofs
G.1 Proof of Theorem 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
G.2 Proof of Lemma 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
G.3 Proof of Theorem 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
G.4 Nonequivalent Condition.
G.5 Failure Condition of Function ρ . . . . . . . . . . . . . . . . . . . . . . . . . . . .
G.6 Proof of Corollary 1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
G.7 Spectral Range of Laplacian Matrix . . . . . . . . . . . . . . . . . . . . . . . . .
G.8 Proof of Corollary 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
I Details of Experiments
I.
Hyper
-parameter Strategy
I.
Baselines of Spectral Convolutions . .
I.
Citation Network Classiﬁcation . .
I.
Visual Object Classiﬁcation . .
I.
Protein Quality Assessment and Fold Classiﬁcation . . . . . . . . . . . . . . . . .
I.
Over-Smoothing Analysis. .
I.
Sensitivity Analysis . . . .
I.
The Spectrum Analysis of SHSC . . . . . . . . . . . . . . . . . . . . . . . . . . .
I.
Ablation Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
I.10 Examples of ρ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
I.11 Running Time and Computational Complexity . . {+. . . . . . . . . . . . . . . . . . I.
+}
The edge-dependent vertex weights visualization . . . . . . . . . . . . . . . . . .
I.13 Additional Experiments of Baselines. . . . . . . . . . . . . . . . . . . . . . . . .
A R ELATED W ORK A.
S IMPLIFIED R ELATED W ORK
Deep learning for hypergraphs.
For spectral-based methods, Feng et al.
(2019) introduce the ﬁrst hypergraph spectral convolution architecture HGNN, based on the hypergraph Laplacian proposed by Zhou et al.
Yadati et al.
(2019) use a non-linear Laplacian operator (Chan & Liang, 2020) to convert hypergraphs to simple graphs by reducing hyperedges into simple edges with mediators, and take advantage of graph neural networks to learning hypergraphs.
Meanwhile, for spatial-based methods.
Dong et al.
(2020) propose a message-passing method that is divided into two processes from vertices to edges and from edges to vertices.
Wendler et al.
(2020) utilize signal processing theory to deﬁne the convolutional neural network on set functions.
Yadati (2020) and Huang & Yang (2021) generalize the current graph message passing methods to hypergraphs and obtain several generalized models.
Even though hypergraph neural networks have been well studied to process the the edge-independent vertex weights, to the best of our knowledge, we are the ﬁrst to attempt on edge-dependent vertex weights.
Hypergraph random walk and spectral theory.
In machine learning applications, Laplacian is an important tool to represent graphs or hypergraphs.
While random walk-based graph Laplacian has a well-studied spectral theory, the spectral methods of hypergraphs are surprisingly lagged.
A two-step random walk Laplaciain was proposed by Zhou et al.
(2006) for the general hypergraph, and was improved by Chitra & Raphael (2019) and Carletti et al.
On the other hand,
Lu & Peng (2011) and Aksoy et al.
(2020) deﬁne a s -th Laplacian through random s -walks on hypergraphs.
More recently, non-linear Laplacian, as an important tool to represent hypergraphs, has been extended to several settings, including directed hypergraphs (Zhang et al., 2017; Chan et al., 2019), submodular hypergraphs (Li & Milenkovic, 2017), etc.
Equivalency between hypergraphs and undigraphs.
A fundamental problem in the spectral theoretical study of hypergraphs is the equivalency with undigraphs.
Once equipped with equivalency, it is possible to represent hypergraphs with lower-order graphs, thus reducing the difﬁculty of hypergraph learning.
So far, equivalence is established from two perspectives: spectrum of Laplacian (Agarwal et al., 2006; Hayashi et al., 2020) or random walk (Chitra & Raphael, 2019).
Hypergraph with edge-dependent vertex weights (EDVW-hypergraph)
EDVW-hypergraphs have been widely adopted in machine learning applications, including 3D object classiﬁcation (Zhang et al., 2018), image segmentation (Ding & Yilmaz, 2010), e-commerce (Li et al., 2018a), image search (Huang et al., 2010), hypergraph clustering (Hayashi et al., 2020) etc.
Unfortunately, to the best of our knowledge, existing learning algorithms for processing EDVWhypergraphs do not involve deep learning methods.
A. D ETAILED R ELATED W ORK
Deep learning for hypergraphs Feng et al.
(2019) introduce the ﬁrst hypergraph architecture hypergraph neural network (HGNN), based on the hypergraph Laplacian proposed by Zhou et al.
(2006), however, a ﬂaw in the theory makes the learning of unconnected networks very inefﬁcient (details can see Appendix D.1).
Yadati et al.
(2019) use the non-linear Laplacian operators Chan & Liang (2020) to convert hypergraphs to simple graphs by reducing a hyperedge to a subgraph with edge weights related only to its degree, which causes the information loss of hypergraphs and limited the generalization.
Jiang et al.
(2019) proposed a dynamic hypergraph neural networks to update hypergraph structure during training.
Wendler et al.
(2020) utilize signal processing theory to deﬁne the convolutional neural network on set functions.
Zhang et al.
(2020) combined hypergraph label propagation with deep learning and introduced a Hypergraph Label
Propagation Network to optimize the hypegraph learning.
Yadati (2020) proposed a generalisedMPNN on hypergraph which uniﬁes the existing MPNNs Gilmer et al.
(2017) on simple graph and also raised a MPNN-Recursive framework for recursively-structured data processing, but it performs poorly for other high-order relationships.
Huang & Yang (2021) generalized the current graph message passing methods to hypergraphs and obtain a UniGCN with self-loop and a deepen hypergrpah network UniGCNII.
However, it does not give reasons why self-loop can assist UniGCN to gain the improvement of performance in the citation network benchmark.
Moreover, its experimental results with various depths may show that the HGCNNs have over-smoothing issue, but theoretical analysis to explain this phenomenon is lacked.
Hypergraph deep learning has also been applied to many areas, such as NLP (Ding et al., 2020), computer vision (Jin et al., 2019), recommendation system (Zhang et al., 2019), etc
Hypergraph random walk and spectral theory.
In machine learning applications, Laplacian is an important tool to represent graphs or hypergraphs.
While random walk-based graph Laplacian has a well-studied spectral theory, the spectral methods of hypergraphs are surprisingly lagged.
The research of hypergraph Laplacian can probably be traced back to Chung (1993), which deﬁnes the Laplacian of k-uniform hypergraph (each hyperedge contains the same number of vertices).
Then Zhou et al.
(2006) deﬁned a two-step random walk-based Laplacian for general hypergraphs.
Based on it, many works try to design a more comprehensive random walk on hypergraphs.
Chitra & Raphael (2019) designed a random walk which considered edge-dependent vertex weights of hypergraphs in the second step to replace the edge-independent weights in Zhou et al.
Actually, the edge-dependent vertex weights could model the contribution of vertex v to hyperedge e and were widely used in machine learning such as 3D object classiﬁcation (Zhang et al., 2018;
Feng et al., 2019) and image segmentation (Ding & Yilmaz, 2010) etc., but without spectral guarantees.
On the other hand, Carletti et al.
(2020; 2021) takes another perspective to gain more ﬁne-grained information from a hypergraph by taking into account the degree of hyperedges to measure the importance between vertices in the ﬁrst step .
However, a comprehensive random walk should consider the above two aspects at the same time.
Furthermore, different from the two-step random walk
Laplacian of Zhou et al.
(2006), Lu & Peng (2011) and Aksoy et al.
(2020) deﬁne a s -th Laplacian through random s -walk on hypergraphs.
More recently, non-linear Laplacian, as an important tool to represent hypergraphs, has been extended to several settings, including directed hypergraphs (Zhang et al., 2017; Chan et al., 2019), submodular hypergraph (Li & Milenkovic, 2017) etc.
Meanwhile, liner Laplacian has also been developed.
For example, Chitra & Raphael (2019) use a two-step random walk to develop a spectral theory for hypergraphs with edge-dependent vertex weights, and a two-step random walk on hypergraph proposed by Carletti et al.
(2020; 2021) also designed a twostep random walk by involving the degree of hyperedges in the ﬁrst step and lead a linear Laplacian.
In this work, we design a uniﬁed two-step random walk framework to study the linear Laplacians.
Equivalency between hypergraphs and undigraphs.
A core problem in the spectral theoretical study of hypergraphs is the equivalency between hypergraphs and graphs.
Once equipped with equivalency, it is possible to represent hypergraphs with lower-order graphs, thus reducing the difﬁculty of hypergraph learning.
So far, equivalence can be established from two perspectives: spectrum of Laplacian or random walks.
For the aspect of Laplacian spectrum, Agarwal et al.
(2006) ﬁrstly showed the Laplacian of Zhou et al.
(2006) is equivalent to the Laplacian of the corresponding star expansion graph (having the same eigenvalue problem).
Next, Hayashi et al.
(2020) claims that the
Laplacian deﬁned by Chitra & Raphael (2019) is equal to an undigraph Laplacian, but the undi- graph Laplacian needs to be constructed by invoking the stationary distribution, which restricts its application.
Meanwhile, from the random walk perspective, Chitra & Raphael (2019) claims that hypergraph is equivalent to its clique graph (undirected) when the vertex weight of hypergraph is edge-independent.
However, despite the edge-dependent weights were widely used (Ding & Yilmaz, 2010; Zeng et al., 2016; Zhang et al., 2018), the equivalency problem remains open when the vertex weight is edge-dependent.
Protein learning.
Proteins are biological macromolecules with spatial structures formed by folding chains of amino acids, which have an important role in biology.
A protein has a three-level structure: primary, secondary, and tertiary, where primary (sequence of amino acids) and tertiary (3D spatial) structures are often used to model proteins for representation learning.
Based on amino acids sequence, there exist many related works of protein learning, such as Rao et al.
; Bepler &
Berger (2018); Strodthoff et al.
; Conover et al.
(2019), etc.
Meanwhile, many 3D structure based models are also raised for protein learning, such as Olechnovivc & Venclovas (2017);
Derevyanko et al.
; Baldassarre et al.
(2020); Diehl (2019); Baldassarre et al.
; Gligorijevic et al.
Hermosilla et al.
(2021), etc.
However, to the best of our knowledge, despite the hypergraphs have been developed to model proteins (Maruyama et al., 2001), there is still no related work to design the EDVW-hypergraph-based protein learning algorithm.
B R ELATIONS WITH P REVIOUS R ANDOM W ALKS ON H YPERGRAPH B. A C LASSICAL R ANDOM WALK ON H YPERGRAPH
It can be traced back to Zhou et al.
(2006) in which they have given a view of random-walk to analyze the hypergraph normalized cut.
The transition probability of Zhou et al.
(2006) from current vertex u to next vertex v is denoted as:
[Equation]
It is easy to ﬁnd P ( u, v ) means: (i) choose an arbitrary hyperedge e incident with u with probability w ( e ) /d
( u ) where d ( u ) =
(cid:80) e ∈E w ( e ) H ( u, e ) ; (ii) Then choose an arbitrary vertex v ∈ e with probability 1 /δ
( e ) where δ ( e ) =
(cid:80) v ∈V H ( v, e ) , which means to select vertex randomly in the hyperedge.
B. I NTUITION AND A NALYSIS OF T HE U NIFIED R ANDOM W ALK ON H YPERGRAPH
In this subsection, we would like to explain our intuition of developing the uniﬁed random walk framework in Deﬁnition 1 from a view of two-step random walk on a hypergraph.
To generalize the notation of hypergraph random walk in the seminal paper Zhou et al.
(2006), in this work, we tend to explain the process from another perspective.
The fact in Zhou et al.
that
[Equation]
∈E
[Equation]
e ∈E (cid:88) v ∈V
[Equation] [Equation]
u
[Equation]
( v
[Equation]
e
[Equation]
δ ( e
[Equation]
= (cid:88) v ∈V (cid:88)
e ∈ E ( u
[Equation]
v ) w ( e
[Equation]
δ ( e
[Equation]
(10) and
P ( u, v ) =
(cid:88)
e ∈E w ( e ) H ( u, e ) d ( u ) · H ( v, e ) δ ( e ) =
d ( u ) (cid:88) e ∈ E ( u,v ) w ( e ) δ
( e ) = (cid:80) e ∈ E ( u
[Equation]
v )
[Equation]
show that the probability P ( u, v ) also means a normalized weighted adjacency relationship between u and v (Here, E ( u, v ) denotes the hyperedges contained vertices u and v .).
Furthermore, the relationship is actually measured by the sum of w ( e ) δ ( e ) for all e ∈ E concluding pair { u, v } , which demonstrates that a hyperedge e with larger degree ( δ ( e ) ) linked to { u, v } contribute less to the transition probability between u and v .
From Carletti et al.
(2021), which proposed a random walk on hypergraphs (a specital case of our uniﬁed random walk with Q 1 = Q 2 = H and ρ ( · )
= ( · ) σ ), we further explain the intuition of our purpose to design the P ( u, v ) of our uniﬁed random walk.
When Q 1 = Q 2 = H , P ( u, v ) in Eq.can be expressed as:
[Equation]
That is to say, the inﬂuence of hyperedge degree should not be limited to an inverse relationship by introducing the function ρ ( · ) .
Thus, by evolving more ﬁne-grained vertexes weights Q 1 ( u, e ) and
Q 2 ( v, e ) to replace H ( v, e ) and H ( u, e ) , we obtain the ﬁnal transition probability of our uniﬁed hypergraph random walk in equation 3.
B. S PECIAL C ASES OF T HE U NIFIED R ANDOM W ALK F RAMEWORK AND L APLACIAN
We show the existing random walks and Laplacians are the special cases of our uniﬁed random walks and our uniﬁed Laplacians, respectively.
Under our framework, the existing random walk models can be seen as a special case with speciﬁc p 1 , p 2 .
It is easy to construct the relations of our framework to previous works as follows:
Remark 1 (Zhou et al.
When we choose ρ ( · ) = ( · ) − 1 and Q 1 = Q
= H in our framework, the probabilities of uniﬁed random walk on hypergraph are reformulated as p 1 = w ( e ) H ( u,e ) (cid:80) e ∈E w ( e ) H
( u,e ) and p 2 =
H ( v,e ) (cid:80) v ∈V H ( v,e ) .
As the special case of ours satisfying Q 1 = Q 2 = H and ρ ( · ) = ( · ) − 1 , Zhou et al.
(2006)’s random walk satisﬁes both condition (1) and condition (2) in Thm.
So we obtain from Corollary 1 that π zhou ( v ) =
d ( v ) (cid:80) u ∈V d ( u ) and L zhou =
I − D − 1 / 2 v
HWD
[Equation]
where D v ( u, u ) =
d ( u ) =
(cid:80) e ∈E w ( e ) H ( u, e ) and D e ( e, e ) =
δ ( e ) = (cid:80) v ∈ e H ( v, e ) .
The forms of π zhou and L zhou are exactly the same in Zhou et al.
Remark 2 (Carletti et al.
When we select ρ ( · ) = ( · ) σ and Q 1 = Q 2 =
H , the probabilities of uniﬁed random walk on hypergraph are reduced to p= w ( e ) H ( u,e )( (cid:80) v ∈V H ( v,e )) σ +1 (cid:80) e ∈E w ( e ) H ( u,e )( (cid:80) v ∈V H ( v,e )) σ +1 and p 2 =
H ( v,e ) (cid:80) v ∈V H ( v,e ) .
As the special case of ours satisfying
Q 1 = Q 2 = H and ρ ( · )
= ( · ) σ , Carletti et al.
(2021)’s random walk satisﬁes both condition (1) and condition (2) in Thm.
So we obtain from Corollary 1 that π car ( v ) =
d ( v ) (cid:80) u ∈V d ( u ) and L car =
I − D − 1 / 2 v
[Equation]
H (cid:62) D − 1 / 2 v where D v ( u, u ) =
d ( u ) =
(cid:80) e ∈E w ( e ) δ ( e ) σ +1 H ( u, e ) and D e ( e, e )
= δ ( e ) = (cid:80) v ∈ e H ( v, e ) .
The forms of π car and L car are exactly the same in Carletti et al.
Note that the random walk proposed by Carletti et al.
(2020) is a special case of
Carletti et al.
(2021) with σ = 1 , the π and L are easy to obtain from our conclusions.
Remark 3 (Chitra & Raphael (2019))
When we set ρ ( · ) = ( · ) − 1 and Q 1 = H , the probabilities of uniﬁed random walk on hypergraph become p 1 = w ( e ) H ( u,e ) (cid:80) e ∈E w ( e ) H
( u,e ) and p 2 =
Q 2 ( v,e ) (cid:80) v ∈V Q 2 ( v,e ) with an edge-dependent vertex weights Q 2 in the second steps.
Actually, as the special case of ours satisfying Q 1 = H and ρ ( · ) = ( · ) − 1 , Chitra & Raphael (2019)’s random walk on hypergraph generates the Laplacian L chi which could build up a simple relation with our L rw in equation 20 as
[Equation]
which means that L rw denotes a form of symmetrization on L chi .
Further, as the vertex weights is edge-independent in Chitra & Raphael (2019) which means they satisﬁes condition(1) in Thm.
1, the same π as Chitra & Raphael (2019) can be obtained from Corollary 1, i.e. π ( v ) =
ˆ d ( v ) (cid:80) v ˆ d ( v ) .
C U NIFIED N ON - LAZY R ANDOM W ALKS ON H YPERGRAPH
The non-lazy version is nontrivial to factor and analyze.
To ease the studies, we ﬁrst give the following deﬁnition and lemma.
Deﬁnition 4 (Reversible Markov chain)
Let M be a Markov chain with state space X and transition probabilities P ( u, v ) , for u, v ∈ X .
We say M is reversible if there exists a probability distribution π over X such that
[Equation]
Lemma 2 (Chitra & Raphael (2019))
Let M be an irreducible Markov chain with ﬁnite state space S and transition probabilities P ( u, v ) for u, v ∈ S .
M is reversible if and only if there exists a weighted, undirected graph G with vertex set S such that a random walk on G and M are equivalent.
Proof.
Let π be the stationary distribution of M (suppose that is irreducible).
Note that π ( u ) (cid:54) =
due to the irreducibility of M .
Let G be a graph with vertices S .
Different from Chitra & Raphael (2019), we set the edge weights of G to be
[Equation]
where c > 0 is a constant for normalizing π .
With reversibility, G is well-deﬁned (i.e. ω ( u, v ) =
ω ( v, u ) ).
In a random walk on G , the transition probability from u to v in one time-step is: ω ( u,v )
[Equation]
=
P ( u, v ) , since (cid:80) w ∈S P ( u, w ) = 1 .
Thus, if M is reversible, recall Deﬁnition 3, the stated claim holds.
The other direction follows from the fact that a random walk on an undirected graph is always reversible (Aldous & Fill,[70]).
Next, we generalize the non-lazy random walk of Carletti et al.
(2021) to our uniﬁed non-lazy random walk with vertex weights Q 1 , Q 2 .
Deﬁnition 5 (Uniﬁed Non-lazy Random Walk on Hypergraphs) Suppose δ (
[Equation]
:= (cid:80) v ∈V Q 2 ( v, e ) is the degree of hyperedge, and d nl ( v ) :
= (cid:80) e ∈E w ( e )( δ ( e ) − Q 2 ( v, e ))
ρ ( δ ( e ) −
Q 2 ( v, e ))
Q 1 ( v, e ) is the degree of vertex v .
The uniﬁed non-lazy random walk on a hypergraph H is deﬁned in a two-step manner:
Given the current vertex u ,
Step I : choose a hyperedge e incident to u , with probability
[Equation]
Step II : choose an arbitrary vertex v (cid:54) =
u from e , with probability
[Equation]
Thus, the transition probability of our uniﬁed non-lazy random walk from vertex u to v is:
P nl
[Equation]
u
[Equation]
if u (cid:54) =
v
[Equation] [Equation]
u
[Equation]
v (16)
Thus, the transition matrix of uniﬁed non-lazy random walk can be expressed as:
[Equation]
(17) where D nl , D e are diagonal matrices with entries D nl ( v, v ) =
d nl ( v ) and D e ( e, e ) =
δ ( e ) , respectively.
(cid:12) denotes the Hadamard product and 1 denotes a |V|×|E| ones matrix(i.e.
all elements equal to 1).
ρ ( 1D e − Q 2 )) represents the function ρ acting on each element of ( 1D e − Q 2 ) .
Following Chitra & Raphael (2019), the modiﬁed version of the clique graph without self-loops is deﬁned below:
Deﬁnition 6 (Chitra & Raphael (2019)) Let H ( V , E , Q ) be a hypergraph associated with the uniﬁed non-lazy random walk in Deﬁnition 5.
The clique graph of H without self-loops, G Cnl , is a weighted, undirected graph with vertex set V , and edges E (cid:48) deﬁned by
[Equation]
[Equation]
Different from the lazy random walk, a uniﬁed non-lazy random walk on a hypergraph with condition (1) or condition (2) in Thm.
1 is not guaranteed to satisfy reversibility (see Deﬁnition 4).
However, if Q 1 = Q 2 = H , then reversibility holds, and we obtain the result below.
Theorem 2 Let H ( V , E , Q ) be a hypergraph associated with the uniﬁed non-lazy random walk in
Deﬁnition 5.
Let Q 1 = Q 2 = H , i.e. Q i ( v, e ) = 1 , i ∈ { 1 , 2 } for all vertices v incident hyperedges e .
Then, there exist weights ω ( u, v ) on the clique graph without self-loops G Cnl such that a non-lazy random walk on H is equivalent to a random walk on G Cnl .
Proof.
We ﬁrst prove that the uniﬁed non-lazy random walk on H is reversible.
It is easy to verify the stationary distribution of the uniﬁed non-lazy random walk on H with Q 1 = Q
= H is π ( v ) =
[Equation]
Let P ( u, v ) be the probability of going from u to v in a uniﬁed non-lazy random walk on H , where u (cid:54) = v .
Then,
[Equation]
So the uniﬁed non-lazy random walk is reversible.
Thus, by Lemma 2, there exists a graph G with vertex set V and edge weights
[Equation]
such that a random walk on G and the uniﬁed non-lazy random walk on H is equivalent.
The equivalence of the random walks implies that P ( u, v ) >
if and only if ω ( u, v ) > 0 , so it follows that G is the clique graph of H without self-loops.
[Figure]
D T YPICAL H YPERGRAPH C ONVOLUTION (HGNN) AND I TS S PECTRAL A NALYSIS
Recall Remark 1, the random walk and Laplaican proposed by Zhou et al.
(2006) are special cases ( ρ ( · ) =
( · ) − 1 , Q 1 = Q 2 = H ) of our uniﬁed random walk and uniﬁed Laplacian, respectively.
Thus, now the degree of a vertex v ∈ V is d ( v ) =
(cid:80) e ∈E w ( e ) H ( v, e ) and for a hyperedge e ∈ E , its degree is δ ( e ) =
(cid:80) v ∈V H ( v, e ) .
The edge-degree matrix and vertex-degree matrix can be denoted as
D. T YPICAL S PECTRAL H YPERGRAPH C ONVOLUTION
From the seminal paper Zhou et al.
(2006), a classical symmetric normalized hypergraph
Laplacian is deﬁned from the perspective of spectral hypergraph partition:
L =
I −
[Equation]
Based on it, Feng et al.
(2019) follow Kipf & Welling (2017) to deduce the Hypergraph Neural Network HGNN.
Speciﬁcally, they ﬁrst perform an eigenvalue decomposition of L : L = U Λ U (cid:62) , where Λ is a diagonal matrix of eigenvalues and U is consisted of the corresponding regularized eigenvectors.
Combining the hypergraph Laplacian with Fourier Transform of graph signal processing, they derive the primary form of hypergraph spectral convolution: g ∗ x = U (( U (cid:62) g ) (cid:12) ( U (cid:62) x ))
=
U g (Λ) U (cid:62) x
where g (Λ) =
diag ( g ( λ 1 ) , g ( λ 2 ) , · · · , g ( λ N )) is a function of the eigenvalues of L .
To avoid expensive computation of eigen decomposition (Defferrard et al., 2016), they use the truncated Chebyshev polynomials T k ( x ) to approximated g (Λ) , which is recursively deduced by T k ( x ) = 2 xT k − 1 ( x ) − T k − 2 ( x ) with T 0 ( x ) = 1 and T 1 ( x ) = x .
g ∗ x ≈ (cid:80) K − 1 k =
θ k T k
(ˆΛ) x, with scaled Laplacian ˆΛ = 2 λ max Λ − I limiting the eigenvalues in [ − 1 , 1] to guarantee requirement of the Chebyshev polynomial (Hammond et al., 2011).
Since the parameters in the neural network could adapt to the scaled constant ( λ max ≈ 2 ), the convolution operation is further simpliﬁed to be g ∗ x ≈
[Equation]
However, in order to obtain the ﬁnal convolutional layer similar to Kipf & Welling (2017) proposed, Feng et al.
(2019) used a speciﬁc matrix to ﬁt the scalar parameter θ 0 , causing a ﬂaw in the derivation.
In the end, they have managed to obtain a seemingly correct hypergraph spectral convolution (HGNN):
[Equation]
where Θ is the parameter to be learned during the training process.
There is no doubt that HGNN can work properly, but it could have low efﬁciency when the hypergraph of the datasets is disconnected.
Our experiments in subsection 4.1 and Figure 4 verify our conclusion.
D. S PECTRAL A NALYSIS OF HGNN
We give the deﬁnition of Rayleigh quotient to study the spectrum of Laplacian.
Lemma 3 ( Rayleigh Quotient (Horn, 1986)) Assume that M ∈ R n × n is a real symmetric matrix and x ∈ R n is an arbitrary nonzero vector.
Let λ 1 ≤ λ 2 ≤ · · · ≤ λ n denote the eigenvalues of M in order.
Then the Rayleigh quotient satisﬁes the property:
[Equation]
The equality holds that R ( M , u 1 )
=
λ 1 and R ( M , u n ) =
λ n where u i is the eigenvector related to λ
i , i ∈ { 1 , n } .
The Lemma 3 is mainly used to get all exact or approximated eigenvalues and eigenvectors of M .
In this part, we utilize it to prove that the smallest eigenvalue λ min of L =
I −
D − 12 v HWD − 1 e H (cid:62) D − 12 v is 0 and ﬁgure out the corresponding eigenvector u min is D 12 v 1 .
Speciﬁcally, let g denote an arbitrary column vector which assigns to each vertex v ∈ V a real value g ( v ) .
We demonstrate R ( L , g ) as below in order to describe our conclusion distinctly:
[Equation]
where g = D 1 / 2 v f and f ∈ R |V|× 1 .
It is easy to see the fact: R ( L , g ) ≥ 0 , and R ( L , g )
=
if and only if f = 1 (i.e. g = D 1 / 2 v 1 ).
Then with lemma 3 we get λ min =
min g R ( L , g ) =
R ( L , D 1 / 2 v 1 )
= 0 and u min =
D 12 v 1 .
Actually, since the Laplacian of Zhou et al.
(2006) is our uniﬁed Laplacian in Corollary 1, the property above can be directly led by Thm. 5.
E T HE R ELATIONSHIP BETWEEN G ENERALIZED H YPERGRAPHS AND D IGRAPHS
In this part, we would like to illustrate that our uniﬁed random walk on a hypergraph is always equivalent to a random walk on the weighted directed clique graph, As long as the equivalent conditions are established, the graphs can be viewed as the low-order encoders of hypergraphs, and many machine learning methods of digraphs could be easily generalized to hypergraphs.
Chitra & Raphael (2019), whose random walk is a special case of our framework with ρ ( · ) =
( · ) − 1 , Q 1 = H , claims that the hypergraph with edge-dependent weights (i.e. Q 2 is edgedependent) is equivalent to a directed graph.
Here we extend this conclusion to a more general version by the above deﬁnition.
Theorem 3 (Equivalency between hypergraph and weighted digraph)
Let H ( V , E , W , Q 1 , Q 2 ) denote the generalized hypergraph in Deﬁnition 2.
There exists a weighted directed clique graph G C such that the random walk on H is equivalent to a random walk on G C .
Thm. 3 indicates that any hypergraph is equivalent to a digraph under our framework (Fig. 1) and implies that the existing hypergraph random walks are modeling pairwise relationships (proof in
Appendix G.1 ).
Adopting results of Thm.
3, we design the Laplacian for our uniﬁed random walk framework and explore its spectral properties.
Thm. 3 says that the uniﬁed random walk framework is equivalent to directed graph, so we extend the deﬁnition of directed graph Laplacian (Chung, 2005) to obtain the normalized Laplacian matrix as follows.
Corollary 3 ( Random Walk-based Hypergraph Laplacian ) Let H ( V , E , W , Q 1 , Q 2 ) be the generalized hypergraph Let P be the transition matrix of our random walk framework on H with stationary distribution π
(suppose that exists).
Let Φ be a |V| × |V| diagonal matrix with Φ( v, v )
=
π ( v ) .
The uniﬁed random walk-based hypergraph Laplacian matrix L rw is deﬁned as:
[Equation]
Following Chung (2005), it is easy to study many properties based on Rayleigh quotient of L rw , such as positive semi-deﬁnite, Cheeger inequality, etc.
The Laplacian L rw has a great expressive power thanks to the comprehensive random walk, which uniﬁes most of existing random walkbased Laplacian variants due to the generalization ability of the framework (see Appendix B.3).
Note that the distribution π is not easy to obtain and do not admit a unitized expression that limits the application of L rw in equation 20.
In this paper, we focus on the case when hypergraph is equivalent to undigraph (Thm. 1) and deduce the explicit expression of L rw (Corollary 1).
Based on our equivalency Theorem and Lapalacian, we can design directed graph neural networks for generalized hypergraph learning, which we will do as a future work.
F D ERIVATION AND A NALYSIS OF GHCN AND SHSC F. D ERIVATION OF GHCN
We deﬁne K as Q 2 W ρ ( D e ) Q (cid:62) 2 .
Then the uniﬁed hypergraph Laplacian matrix L in Thm.
1 can be denoted as
[Equation]
[Equation] Similar to GCN (Kipf & Welling, 2017) and Appendix D.1, we perform eigenvalue decomposition of L , then obtain the convolution kernel by using Fourier transform, and approximate it with the truncated Chebyshev polynomial to get the convolution form:
[Equation]
Then we adopt the same method proposed by Kipf & Welling (2017) to set the parameter θ
= θ 0 = − θ 1 and obtain the following expression:
[Equation]
According to Theorem 5, it is easy to know the eigenvalues of I + ˆ D − 1 / 2 v
[Equation]
range in [0 , 2] , which may cause the unstable numerical instabilities and gradient explosion problem.
So we use the renormalization trick (Kipf & Welling, 2017):
[Equation]
where ˜ K
= K
+ I can be regarded as the weighted adjacency matrix of the equivalent weighted graph with self-loops, and ˜ D ( v, v ) =
(cid:80) u
˜ K ( v, u ) is a |V| × |V| diagonal matrix.
This self-loops makes the methods more robust to process disconnected hypergraphs datasets.
Finally, we drive the generalized hypergraph convolutional network( GHCN ):
[Equation]
Θ is a learnable parameter and ψ ( · ) denotes an activation function.
F. D ERIVATION OF T HE D ISCOUNTED M ARKOV D IFFUSION K ERNEL
Discounted Markov Diffusion Kernel for hypergraph Recall that the discounted average visiting rate :
[Equation]
i.e. Z ( t ) =
(cid:80)
[Equation]
P
[Equation]
Then, ¯ v i ( t ) =
(¯ v i 1 ( t ) , · · · , ¯ v in ( t )) (cid:62) denotes as:
[Equation]
Recall the deﬁnition of ¯ v i (0) in subsection 3.2: ¯ v ik (0)
=
if and only if i = k , otherwise ¯ v ik (0) = 0 .
Then, the discounted Markov diffusion kernel K MD can be derived from the l 2 -norm diffusion distance as follows:
[Equation]
F. D ERIVATION OF SHSC
Inspired by the diffusion kernel-based methods from Zhu & Koniusz (2021) and Klicpera et al.
(2019), and based on the discounted Markov diffusion kernel deﬁned in subsection 3.2, the hypergraph convolution designed in our work is shown as below general form:
[Equation]
where ˜ T is the generalized transition matrix in equation 24, Θ is the parameter matrix to be learned during training, and ψ ( · ) denotes an activation function.
Since ˜ T k can be viewed as a k -steps transition matrix, in order to weaken the neighbor vertices in long diffusion steps, we introduce a discounting factor α k K for ˜ T k , which can be deduced from the underlying feature map Z ( t ) of discounted Markov diffusion kernel deﬁned in subsection 3.2.
We use a well-known trick (Zhu &
Koniusz, 2021; Chen et al., 2020) to balance the global information aggregation and the vertex’s own information by introducing a hyper-parameter β ∈ [0 , 1] to control the self-loop ˜ T
= I .
Finally, we proposed a Simple Hypergraph Spectral Convolution ( SHSC ) as:
[Equation]
Actually, SHSC is a spatial-based model while we can analyze it from a spectral-based view.
Let
L = I − ˜ T denote the normalized Laplacian matrix.
Then the SHSC can be viewed as a special spectral-based polynomial ﬁlter through (cid:80) Ii =
ξ i L i =
(cid:80) Kk =
θ k ˜ T k where θ k = α k K .
Finally, we can deduce the coefﬁcients of the special polynomial ﬁlter as ξ i
=
( − 1)
i (cid:80)
Kk
= i (cid:0) ki (cid:1)
α k K showing the strong relationships between SHSC and spectral-based model.
The explicit form of ξ i can be derived as:
[Equation]
(cid:33)
L i
F. O VER - SMOOTHING A NALYSIS OF GHCN AND SHSC
For real-world datasets, we always use non-Euclidean graphs with ﬁnite-dimensional features of vertices.
It is difﬁcult to use the absolute diffusion distance (Masuda et al., 2017) which measures the distance walked from starting vertex in the diffusion process.
Therefore, we use the reciprocal of the mean l 1 -norm error between t -step transition probability P t and stationary distribution π as the measure of over-smoothing in arbitrary starting vertex.
Formally, we name the measure by over-smoothing energy as follows:
[Equation]
where i denotes the sign of starting vertex i and t denotes the number of diffusion step.
As mentioned in Corollary. 2, f ( i ) is an initial distribution which means the walker diffuse starting from vertex
i (i.e. f ( i ) i = 1 and f ( i ) j = 0 , ∀ j (cid:54) = i ).
It is easy to observe that if the limit of e ( i, t ) with respect to t converges to inﬁnity, it indicates the model based on P undergo severe over-smoothing issue.
Recall Corollary 2, we get a low-bounded quantity of e ( i, t ) as e low ( i, t ) :
[Equation]
where ϕ ( H ) =
(cid:80) Nj =1 (cid:113)
ˆ d ( j ) can be seen as a constant associated with H which is independent with i or t .
Recall Corollary 2 that λ H is smallest nonzero eigenvalue of L .
Over-smoothing analysis of GHCN.
As shown in Table 14, we can see our proposed GHCN model also face with the over-smoothing phenomenon.
Assume that we get a K -layers GHCN model with the form: X ( l +1) =
ReLU(˜ TX ( l ) Θ ) .
To get a simple mathematical form from above incremental convolution formula (remove the activation layer), we could get approximated K -layers
GHCN as:
[Equation]
Intuitively, from the simple form we ﬁnd the over-smoothing energy of GHCN in vertex i can be represented by e low ( i, K ) .
It is easy to analyze that as K → ∞ , e low ( i, K ) converge to inﬁnity which implies that the over-smoothing energy e ( i, t ) converge to inﬁnity.
Furthermore, the convergence of over-smoothing energy to inﬁnity means the error between initial signal and π converge to 0 as the number of layers increasing, which describes the reason of over-smoothing underwent by GHCN from a quantitative perspective.
HGNN (Feng et al., 2019) also suffers from over-smoothing as shown in Table 14.
Meanwhile, the theoretical analysis of over-smoothing on HGNNcan be covered by the analysis of GHCN due to HGNN is a special case of GHCN.
Over-smoothing analysis of SHSC.
Similarly, from the form of SHSC, we use
K (cid:80) Kk
=
α k e low ( i, k ) to represent SHSC’s over-smoothing energy .
In order to explain the reason why SHSC could relieve over-smoothing, we provide the idea of equivalent inﬁnitesimals to compare the over-smoothing energy between muti-layers GHCN and SHSC as follows:
As
K → ∞ :
[Equation]
29) where the last equation depends on α ∈ (0 , 1] and λ H ∈ (0 , 2] proved by Theorem 5.
If 1 − λ H − α = 0 , 1 − λ H = α < 1 .
Thus, the equation 29 also holds.
The result describes why SHSC can relieve over-smoothing compared to multi-layer GHCN that the over-smoothing energy of multilayers GHCN has been sparse by the designed diffusion kernel of SHSC as K growing.
Actually, we can directly analyse the over-smoothing energy of SHSC:
[Equation]
If α ≤ | 1 − λ H | and 1 − λ H − α (cid:54) = 0 , equation 30 equals to 0 ; If 1 − λ H − α = 0 , equationequals to N √ ˆ d
( i ) ϕ
( H ) ;
Otherwise, equation 30 converges to inﬁnity.
Notably, this result shows that the over-smoothing property of SHSC is related to the magnitude of α and λ H .
Moreover, this analysis further reveals that the introduced hyperparameter α is meaningful for SHSC to alleviate over-smoothing.
G D ETAILS OF T HEORIES AND P ROOFS G. P ROOF OF T HEOREM
Theorem 3 (Equivalency between hypergraph and weighted digraph)
Let H ( V , E , W , Q 1 , Q 2 ) denote the generalized hypergraph in Deﬁnition 2.
There exists a weighted directed clique graph G C such that the random walk on H is equivalent to a random walk on G C .
Proof.
Let ω ( u, v ) be the weight of edge ( u, v ) in G C .
We set ω ( u, v ) =
d ( u ) P ( u, v ) in Deﬁnition 1, then the weighted clique graph is a directed graph, where P ( u, v ) is the transition probabilities in Eq equation 3.
The random walk on the directed weighted G C with transition probabilities
[Equation]
is equivalent to H with transition probabilities P ( u, v ) .
The matrix form of edge weight of G C is :
[Equation]
Suppose W =
I and let ρ ( · ) =
( · ) − 1 , we can obtain the equivalent weighted clique graph showing in Figure 5.
[Figure]
G. P ROOF OF L EMMA
Lemma 1 Let
H (
[Equation]
[Equation]
denote the generalized hypergraph in
Deﬁnition 2.
Let F ( Q 1 ,Q 2 ) ( u, v )
[Equation]
= (cid:80) e ∈E w ( e ) ρ ( δ ( e ))
Q 1 ( u, e ) Q 2 ( v, e ) and T ( Q ) ( u ) :
= (cid:80) e ∈E w ( e ) δ ( e ) ρ ( δ ( e ))
Q ( u, e ) .
When Q 1 , Q 2 satisﬁes the following equation
[Equation]
there exists a weighted undirected clique graph G C such that a random walk on H is equivalent to a random walk on G C with edge weights ω ( u, v )
=
T ( Q 2 ) ( u ) F ( Q 1 ,Q 2 ) ( u, v ) / T ( Q 1 ) ( u ) if T ( Q 1 ) ( u ) (cid:54) = 0 and 0 otherwise.
And the symmetrical Laplacian of G C is
[Equation]
where I {·} denotes the indicator function.
Proof.
Form Lemma 2, we just need to prove Markov chain with state space V is reversible.
The transition probabilities of the uniﬁed random walk on H are:
[Equation]
By equation 4, we have:
[Equation]
i.e.
[Equation]
Comparing with equation 12, we normalize G ( Q 2 ) to deﬁne the stationary distribution of M :
[Equation]
(32) which can be easy to verify by (cid:80)
u π ( u ) P [Equation] u
[Equation]
v ) =
π ( v ) .
Thus, π ( u ) P ( u, v ) =
π ( v ) P ( v, u ) , which suggests that M is reversible.
By Lemma 2, the edge weight ω ( u, v ) of G C is
[Equation]
To gain the Laplacian of G C , we calculate the sum of row of edge weights matrix,
[Equation]
Then we get the symmetrical Laplacian of G C :
[Equation]
[Equation]
G. P ROOF OF T HEOREM
Theorem 1 (Equivalency between generalized hypergraph and weighted undigraph) Let
H ( V , E , W , Q 1 , Q 2 ) denote the generalized hypergraph in Deﬁnition 2.
When H satisﬁes any of the condition bellow:
Condition (1) Q 1 and Q 2 are both edge-independent;
Condition (2) Q 1 = k Q 2 ( k ∈ R ) , there exists a weighted undirected clique graph G C such that a random walk on H is equivalent to a random walk on G C .
Proof.
1) Proof based on Lemma
For condition (1),
[Equation]
Substituting the above equations into equation 4, then equation 4 holds.
For condition (2),
[Equation]
Similarly, substituting the above equations into equation 4, then equation 4 holds.
2 ) Proof based on Lemma
I)
For Condition (1):
Because Q 1 and Q 2 are both edge-independent, we set Q 1 ( u, e ) =
q 1 ( u ) and Q 2 ( v, e ) =
q 2 ( v ) for all hyperedge e .
From Lemma 2, the key is to prove the uniﬁed hypergraph random walk on H under condition (1) is reversible (see Deﬁnition 4).
It is hard to ﬁnd the explicit form of π before we get Corollary 4.
Fortunately, by Kolmogorov’s criterion, that is equal to prove:
[Equation]
for arbitrary subset { v 1 , · · · , v n } ⊆ V .
The transition probabilities of the generalized random walk on H are:
[Equation]
where E ( u, v ) = { e ∈ E : u ∈ e, v ∈ e } to be the set of hyperedges incident to both v and u .
Then we have:
[Equation]

 , where set v n +1 = v
[Equation]
So the uniﬁed random walk on H under condition (1) is reversible.
As a matter of fact, we can succinctly obtain the reversibility via the stationary distribution.
Specifically, we can verify the stationary distribution is
[Equation]
[Equation]
by (cid:80) u π ( u ) P ( u, v ) =
π ( v ) .
Then with π ( u ) P ( u, v ) =
π ( v ) P ( v, u ) holding, we know our uniﬁed random walk on H under condition (1) is reversible.
Since H is connected, the uniﬁed random on
H is irreducible.
From Lemma.2, We know that the current uniﬁed random walk on H is equivalent to a random walk on an undirected graph G with vertex set V .
By equation 13, the edge weights of
G are:
[Equation]
where c = (cid:80) v ∈V (cid:80) e ∈E w ( e ) δ ( e ) ρ ( δ ( e ))
Q 2 ( v, e ) .
Note that P ( u, v ) >
if and only if ω ( u, v ) > 0 , so G is the clique graph of H .
II)
For condition (2) that Q 1 = k Q 2 , we have P = k D − 1 v Q 2 W ρ
( D e ) Q (cid:62) 2 .
Thanks to the underlying symmetry adjacency matrix of
P (i.e. Q 2 W ρ ( D e ) Q 2 is symmetrical), we have: 1 (cid:62) ˆ D v P = 1 (cid:62) ˆ D (cid:62) v .
Thus, the stationary distribution is
[Equation]
Further, we would prove that the uniﬁed random walk on H under current condition (2) is reversible.
[Equation]
So the uniﬁed hypergraph random walk on H under condition (2) is reversible.
Since H is connected, the uniﬁed random on H is irreducible.
From Lemma.2, We know that the current uniﬁed random walk is equivalent to a random walk on a weighted, undirected graph G with vertex set V .
By equation 13, the edge weights of G are:
[Equation]
[Equation]
where c = (cid:80) v (cid:48)
d ( v (cid:48) )
= (cid:80) v ∈V (cid:80) e ∈E w ( e ) δ ( e ) ρ ( δ ( e ))
Q ( v, e ) .
Note that P ( u, v ) >
if and only if ω ( u, v ) > 0 , so G is the clique graph of H .
On the whole, we get the speciﬁc equivalent weighted undigraphs under our condition (1) and condition (2).
Note that they have the same edge weights expression ( equation 37,equation 39):
[Equation]
which further suggests the connection between condition (1) and condition (2) stated in Corollaryand Thm. 1.
Remark: Let ρ ( · ) = ( · ) − 1 and W =
I , we can get the undigraph shown in Figure 1.
[Figure]
N ONEQUIVALENT C ONDITION .
Theorem 4 There at least exists one generalized hypergraph H ( V , E , W , Q 1 , Q 2 ) (Deﬁnition 2) with edge-dependent weights and Q 1 (cid:54) =
k Q 2 , such that a random walk on H is not equivalent to a random walk on its undirected clique graph G C for any choice of edge weights on G C .
Remark.
Theorem 4 tells that the random walk on a hypergraph can not be always equivalent to a random walk on an undigraph, and open up the possibility to construct a hypergraph is not equivalent to an undigraph, which may inspire researcher to further study for higher-order interactions and other insurmountable bottlenecks on hypergraphs.
Proof.
When the vertex weights are edge-dependent, the reversibility is not always satisﬁed in our uniﬁed random walk on a hypergraph.
By Lemma 2, if the uniﬁed random walk is not timereversible, it could not equivalent to random walks on digraphs for any choice of edge weights.
A irreversible example can see Figure 6.
The probability transition matrix of the uniﬁed random walk on the left hypergraph (with W
=
I , ρ ( · ) =
( · ) − 1 ) is given below:
[Equation]
                   , (40) and the stationary distribution of the uniﬁed random walk on hypergraph with P is:
[Equation]
We verify π (1) P (cid:48) (1 , 2) = 3 17 × 1 3 (cid:54) =
7 17 × 1 6 = π (2) P (2 , 1) , which indicates that the Markov chain represented by the uniﬁed random walk in Figure 6 with transition probabilities P is irreversible.
G. F AILURE C ONDITION OF F UNCTION ρ .
As a matter of fact, the ρ in our random walk framework can not always work and we give its failure condition as follows.
Proposition 1 Let H ( V , E , W , Q ) associated with the uniﬁed random walk in Deﬁnition 1.
ρ ( · ) fails to work in our uniﬁed random walk if the hyperedge degrees are edge-independent (i.e. δ ( e )
=
δ ( e (cid:48) ) , ∀ e (cid:48) ∈ E ).
Proof.
Given δ ( e ) are edge-independent, let us suppose δ ( e )
=
k , then
[Equation]
which indicates P ( u, v ) is independent to ρ ( · ) , i,e. ρ ( · ) fails to work.
It is obvious that the k -uniform hypergraph has edge-independent hyperedge degrees.
Formally, we have the following proposition.
Proposition 2 Let H ( V , E , W , Q 1 , Q 2 ) denote the generalized hypergraph in Deﬁnition 2 with ρ ( · ) = ( · ) σ and Q 1 = Q 2 =
H , then H is a k -uniform hypergraph(i.e each hyperedge contains the same number of vertices) if and only if P ( u, v ) in equation 3 is independent with σ .
Proof.
For Q = H , then δ ( e ) =
(cid:80) v ∈ e h ( v, e ) = k .
So, for all u, v ∈ V , we have:
[Equation]
Then we get the entries of the transition matrix P ( σ ) :
[Equation] [Equation]
[Equation]
It is easy to see that P ( σ ) ( u, v ) is independent with σ .
The other direction, we ﬁnd when P ( σ ) ( u, v ) in is independent with σ ,then:
[Equation] [Equation]
=
[Equation] [Equation]
V
[Equation]
It is equal to ∃ k ∈ Z
++ s.t.:
[Equation]
E .i.e.
the hypergraph is k-uniform.
With the condition that Q 1 = Q 2 = H , our uniﬁed random walk on hypergraph reduces to a random walk on hypergraph that leaving only incident relationships between hyperedges and vertices.
Actually, the reduced random walk mentioned is a common structure in real-world applications.
The proposition is to say when the hypergraph based on the reduced random walk is uniform, the function ρ ( · ) does not impact the importance between vertices any more.
G. P ROOF OF C OROLLARY 1.
Before proving the Corollary 1, we further illustrate the fact that Lemma 4 states.
Lemma 4 Let H ( V , E , W , Q 1 , Q 2 ) be the generalized hypergraph in Deﬁnition 2.
Given a H satisfying condition (1) in Thm. 1, there exists a H (cid:48) satisfying the condition (2) such that the random walk on H is equivalent to that on H (cid:48) .
Proof of Lemma 4.
When the condition (2) in Thm.
1 holds (i.e. Q 1 = Q 2 ), the transition probabilities of the uniﬁed random walk on H are
[Equation]
Meanwhile, if the condition (1) in Thm.
1 holds (i.e. Q 1 , Q 2 are both edge-independent), for all hyperedge e , we represent Q 1 ( u, e ) , Q 2 ( v, e ) as q 1 ( u ) , q 2 ( v ) respectively, and the transition probabilities are
[Equation]
It can be observed that P c 1 ( u, v ) =
P c 2 ( u, v ) , ∀ u, v ∈ V .
By deﬁnition 3, the Lemma 4 holds.
Next, based on Lemma 4, we prove the following main conclusion.
Corollary 1 Let H ( V , E , W , Q 1 , Q 2 ) be the generalized hypergraph in Deﬁnition 2.
Let ˆ D v be a |V| × |V| diagonal matrix with entries
ˆ D v ( v, v ) :
= ˆ d ( v ) :
= (cid:80) e ∈E w ( e ) δ ( e ) ρ ( δ ( e ))
Q 2 ( v, e ) .
No matter H satisﬁes condition (1) or condition (2) in Thm.
1, it obtains the uniﬁed explicit form of stationary distribution π
and Laplacian matrix L as:
[Equation]
Proof of Corollary 1.
1) Proof based on Lemma 1 When the generalized hypergraph satisﬁes Theorem 1, we obtain the equation 34 and equation 35 in Appendix G.2.
substituting the equations into equation 33 and equation 32, we can obtain the Corollary 1.
2) Proof based on Lemma
We know whether H satisfying condition (1) or condition (2) in
Theorem 1, there exists a weighted undirected clique graph G C equivalent to H .
Recall Lemma 4, when H satisﬁes any of the two conditions, the probability transition matrix P of H can be denoted as:
[Equation]
Then, we obtain the uniﬁed form of probability transition matrix P under any of the two conditions:
[Equation]
where ˆ D v is a |V| × |V| diagonal matrix with entries ˆ
[Equation]
Remark the P is also the probability transition matrix of the equivalent undigraph in Theorem 1.
Due to the symmetric adjacency matrix underlying P (i.e. Q 2 W ρ ( D e ) Q (cid:62) 2 is symmetrical), it is easy to derive the uniﬁed stationary distribution π by 1 (cid:62) ˆ D v P = 1 (cid:62) ˆ D v .
Thus, the stationary distribution is:
[Equation]
Substitute P and π into equation 20, we ﬁnally get the uniﬁed hypergraph Laplacian L as:
[Equation]
Remark.
The stationary distribution with edge-independent vertex weights in Chitra & Raphael (2019) can be seen as a special case of π in equation 51 by setting Q 1 = H , Q 2 ( u, e )
=
γ ( u ) and ρ ( · ) = ( · ) − 1 :
[Equation]
where d Ch ( u ) =
(cid:80) e ∈E w ( e ) h ( u, e ) is the deﬁnition of vertex degree in Chitra & Raphael (2019).
G. S PECTRAL R ANGE OF L APLACIAN M ATRIX
The following Theorem proves the upper bound for eigenvalues of L which meets the requirement of Chebyshev polynomials to enable the derivation for our proposed GHCN in Appendix F.1 and leads to Corollary .
in the paper.
Theorem 5 Let L
=
I − ˆ D − 1 / 2 v
[Equation]
denote the uniﬁed hypergraph Laplacian matrix in Corollary 1 and λ 1 ≤ λ 2 ≤ · · · ≤ λ n denote the eigenvalues of L in order.
1) λ min = λ 1 = 0 and u
=
ˆ D 12 v 1 (the eigenvector associated with λ 1 ) 2) For k = 2 , 3 , · · · , n , we have
[Equation]
here, S k − 1 is the subspace spanned by eigenvectors { u 1 , · · · , u k − 1 } where u i related to λ
i and β ( e, u, v ) =
w ( e ) ρ ( δ ( e ))
Q 2 ( v, e ) Q 2 ( u, e ) .
3) λ max = λ n ≤
Proof: To analyze the generalized Laplacian proposed in our work, we use the Rayleigh quotient mentioned in the Lemma 3 to give out some conclusions about the eigenvalues and eigenvectors of the Laplacian.
) Deduce the Rayleigh quotient of L 2 :
[Equation]
where g = ˆ D 1 / 2 v f and f ∈ R |V|× 1 .
It is easy to see the fact: R ( L , g ) ≥ 0 and R ( L , g )
=
if and only if f = 1 (i.e. g = ˆ D 1 / 2 v 1 ), Then with Lemma 3 we get λ 1 =
=
From
theorem (Horn, 1986)
we have:
[Equation]
We have the fact:
[Equation]
And from Lemma 3, we get:
[Equation]
, u
[Equation]
G. P ROOF OF C OROLLARY
Corollary 2 Let H ( V , E , W , Q 1 , Q 2 ) be the generalized hypergraph in Deﬁnition 1.
When H satisﬁes any of two conditions in Thm.
[Equation]
let L and π be the hypergraph Laplacian matrix and stationary distribution from Corollary 1.
Let λ H denote the smallest nonzero eigenvalue of L .
Assume an initial distribution f with f ( i ) =
1 ( f ( j ) = 0 , ∀ j (cid:54) = i ) which means the corresponding walk starts from vertex v i .
Let p ( k ) =
fP k be the probability distribution after k steps uniﬁed random walk where P denotes the transition matrix, then p ( k ) ( j ) denotes the probability of ﬁnding the walker in vertex v j after k steps.
We have:
[Equation]
Proof: From equation 50:
[Equation]
Then we have:
[Equation]
Then the stationary distribution π ∈ R 1 ×|V| of P can be denoted as:
[Equation]
where c = 1 (cid:62) ˆ D v 1 = (cid:80) v ∈V ˆ d ( v ) > 0 is the sum of 1 (cid:62) ˆ D v .
Let λ 1 ≤ λ 2 ≤ · · · ≤ λ n denote the eigenvalues of L in order.
Then we assume that f ˆ D − 12 v
[Equation]
denotes the l 2 -norm orthonormal eigenvector related with λ i .
From Thm .5, we know ˜ u
1 = u 1 (cid:107) u 1 (cid:107)
=
ˆ D 12 v 1 √ c .
Then a 1 can be computed as:
[Equation]
Then,
[Equation]
[Table]
H G ENERALIZED H YPERGRAPH P ARTITION
For arbitrary vertex subset S ⊂ V , S c denotes the compliment of S .
We deﬁne the hyperedge boundary ∂S as ∂S := { e ∈ E | e ∩ S (cid:54) = ∅ , e ∩ S c (cid:54) = ∅} .
We will generalize the hypergraph partition problem in Zhou et al.
(2006) to our uniﬁed hypergraph framework.
Hypergraph partition is to cut vertex set V into two parts S and S c .
As shown in Theorem 3, we are inspired to adopt the custom of directed graph normalized cut to formulate the hypergraph partition in a mathematical expression.
We need to reuse the deﬁnition of vol( S ) and vol( ∂S ) for directed graphs as:
[Equation]
And the objective function of the hypergraph partition can be expressed as:
[Equation]
As we need the explicit form of stationary distribution to formulate this problem, we set the uniﬁed random walk on a hypergraph that satisﬁes with the conditions in Thm. 1.
Then we have:
[Equation]
where m ( e ∩ S )
= (cid:80) u ∈ e ∩ S Q 2 ( u, e ) is a measure on the subset of V which explicitly demonstrates the difference between Zhou et al.
(2006) (where m ( e ∩ S ) =
(cid:80) u ∈ e ∩ S H ( u, e ) ) and our work.
We considerate more ﬁne-grained information on vertices when we establish a measure on arbitrary subset of V and involve the degree of hyperedges to impact the corresponding hyperedge weights rather than use the a priori hyperedge weights.
I D ETAILS OF E XPERIMENTS
Note that our GHCN and SHSC are both based on the Laplacian led by the equivalent condition in Thm .1, which means the vertex weights we use in our experimental datasets should satisfy the condition (1) or condition (2) in Thm. 1.
I. H YPER - PARAMETER S TRATEGY
We use grid search strategies to adjust the hyper-parameters of our GHCN and SHSC.
The range of hyper-parameters listed in Table 6, Table 7, Table 8, and Table
[Table]
[Table] [Table]
Extra parameter settings for Visual Object Classiﬁcation include k = 6 .
[Table]
I. B ASELINES OF S PECTRAL C ONVOLUTIONS
For HyperGCN (Yadati et al., 2019), we reproduce it by public code with the data already split ( splits).
But the result in its original paper is via running 1000 times with randomly split to get the best 100 times, which is not very common.
Therefore, despite the HyperGCN is the SOTA method on Table 10, it can not achieve a competitive performance under the 10-splits.
For HGNN (Feng et al., 2019), in addition to the ﬂaw during the derivation (i.e. use a speciﬁc matrix to ﬁt the scalar parameter θ 0 ), when doing the Visual Object Classiﬁcation experiment in their public code, it also takes Gaussian kernel as the incidence matrix for hypergraph learning, which lacks theoretical guarantee.
So in our experiment, we just use the incidence matrix H which only consists of elements { 0 , 1 } as the incidence matrix using in HGNN.
HGNN and HyperGCN we use the same grid search strategy as our method GHCN.
I. C ITATION N ETWORK C LASSIFICATION
Datasets.
The datasets we use for citation network classiﬁcation include co-authorship and cocitation datasets: PubMed, Citeseer, Cora (Sen et al., 2008) and DBLP (Rossi & Ahmed, 2015).
We adopt the hypergraph version of those datasets directly from Yadati et al.
(2019), where hypergraphs are created on these datasets by assigning each document as a node and each hyperedge represents (a) all documents co-authored by an author in co-authorship dataset and (b) all documents cited together by a document in co-citation dataset.
The initial features of each document (vertex) are represented by bag-of-words features.
The details about vertices, hyperedges and features are shown in Table 10. [Table] [Table]
Settings and baselines.
We adopt the same dataset and train-test splits (10 splits) as provided by in their publically available implementation 1 .
Note that this dataset just has the edge-independent vertex weights H , which is also called the incidence matrix.
So this experiment can be regarded as a special case of the speciﬁc application of our model (i
[Table]
e. Q = H ).
For baselines, we include
Multi-layer perceptron (MLP) which can be considered as the model without hypergraph structure, current state-of-the-art message-passing hypergraph framework UniGNN (Huang & Yang, 2021) and two recent spectral-based hypergraph convolutonal networks: Hypergraph neural networks(HGNN)
(Feng et al., 2019) and HyperGCN (Yadati et al., 2019).
For UniGNN, we reuse the results of UniGCN reported in (Huang & Yang, 2021)
[Table]
and for HGNN and HyperGCN, we reproduce them following Appendix I.2.
We use cross-entropy loss and Adam SGD optimizer with early stopping with patience of 100 epochs to train GHCN and SHSC.
For other hyper-parameters, we use the grid search strategy.
It is worth noting that the layer number of our model is K in equation 8.
More details of hyper-parameters can be found in Table 6. [Table]
I. V ISUAL O BJECT C LASSIFICATION
Datasets and Settings.
We employ two public benchmarks: Princeton ModelNet40 dataset (Wu et al., 2015) and the National Taiwan University (NTU) 3D model dataset (Chen et al., 2003), as shown in Table 11.
In this experiment, each 3D object is represented by the feature vectors which are extracted by
Multi-view Convolutional Neural Network (MVCNN)
(Su et al., 2015) or Group-View Convolutional Neural Network (GVCNN)
(Feng et al., 2018).
The features generated by different methods can be considered as multi-modality features.
The hypergraph structure we designed is similar to
Zhang et al.
( but they did not give spectral guarantees for supporting the rationality of their practices).
We represent the hypergraph structure as a edge-dependent vertex weight Q to satisfy the condition (2) in Thm.
1 (i.e. Q 1 = Q 2 = Q ).
Speciﬁcally, we ﬁrstly generate hyperedges by k -NN approach, i.e. each time one object can be selected as centroid and its k nearest neighbors are used to generate one hyperedge including the centroid itself (in our experiment, we set k = 10 ).
Then, given the features of data, the vertex-weight matrix Q is deﬁned as
[Equation]
where d ( v, v c ) is the euclidean distance of features between an object v and the centroid object v c in the hyperedge and ˆ d is the average distance between objects.
γ is a hyper-parameter to control the ﬂatness.
Because we have two-modality features generated by MVCNN and GVCNN, we can obtain the matrix Q { i } which corresponds to the data of the i -th modality ( i ∈ { 1 , 2 } ).
After all the hypergraphs from different features have been generated, these matrices Q { i } can be concatenated to build the multi-modality hypergraph matrix Q =
[ Q { 1 } , Q { 2 } ] .
The features generated by
GVCNN or MVCNN can be singly used, or concatenated to a multi-modal feature for constructing the hypergraphs.
For baselines, we just compare with HGNN method for multi-modality learning, following the settings of Appendix I.2.
We also compare our methods using two-modality features to recent SOTA methods on ModelNet40 dataset.
And we use the datasets provided by its public Code 2 .
We use cross-entropy loss and Adam SGD optimizer with early stopping with patience of 100 epochs to train GHCN and SHSC.
It is worth noting that the layer number of our model is K in equation 8.
More details of hyper-parameters can be found in Table 6.
I. P ROTEIN Q UALITY A SSESSMENT AND F OLD C LASSIFICATION
Protein hypergraph modeling.
At the high level, a protein is a chain of amino acids (residues) that will form 3D structure by spatial folding.
In order to simultaneously consider protein sequence and spatial structure information, we build sequence hyperedge and distance hyperedge.
Speciﬁcally, given a protein with | S | amino acids, we choose τ consecutive amino acids ( v i , v i +1 , · · · , v i + τ ) to connect to form a sequence hyperedge and choose amino acids whose spatial Euclidean distance is less than a threshold (cid:15) > 0 to connect to form a spatial hyperedge, where v i ( i = 1 , · · · , | S | ) represent the i -th amino acid in the sequence.
Let E s and E e denote sequence hyperedge and spatial hyperedge, respectively.
Then, we design an edge-dependent vertex-weight matrix Q for capturing the more granular high-order relationships of proteins below (actually, our models GHCN and SHSC allow one to design a more comprehensive Q for learning proteins better):
[Equation]
where d ( v, v c ) < (cid:15) is the euclidean distance between an amino acid v and the centroid amino acid v c in the hyperedge and
ˆ d v c is the average distance between v c and the other amino acids { v i } i (cid:54)
=
c .
γ is a hyper-parameter to control the ﬂatness.
Here we just design an edge-dependent vertex-weights matrix Q for satisfying the condition (2) in Thm.
1 (i.e. Q 1 = Q 2 = Q ).
Note that this Q matrix pays more attention to the information of the sequence (Figure 7).
Experimental settings In our experiment, we set τ = 6 and (cid:15) = 8˚ A .
The initial node features, following Baldassarre et al.
(2020), are composed of amino acid types and 3D spatial features including dihedral angles, surface accessibility and secondary structure type generated by DSSP (Kabsch & Sander, 1983).
Note that both Quality Assessment and Fold Classiﬁcation include graph-level tasks, which means we should add global pooling layers to readout the node representations from our GHCN or SHSC.
Here, for the sake of simplicity, we adopt the permutation-invariant operator mean pooling and a single layer MLP as Readout layers to obtain a global hypergraph embedding and add the softmax (classiﬁcation) or sigmoid (regression) activation function before output.
[Table]
Protein QA is used to estimate the quality of computational protein models in terms of divergence from their native structure.
It is a regression task aiming to predict how close the decoys to the unknown, native structure.
Inspired by Baldassarre et al.
[Table] we train our models on Global Distance Test Score (Zemla, 2003) [Table] which is the global-level score
[Table]
and the Local Distance Difference
Test (Mariani et al., 2013), an amino-acids-level score.
The loss function of QA is deﬁned as the
Mean Squared Error (MSE) losses:
[Equation]
where P gpred and P lpred denote predicted score of global and local respectively.
I.5. P ROTEIN Q UALITY A SSESSMENT (QA)
[Table] Dataset and settings We use the data from past years’ editions of CASP, including CASP1013.
We randomly split the CASP10, CASP11, CASP13 for training and validation, with ratio training: validation = 9:1.
CASP 12 is set aside for testing against other methods.
More details about the datasets can be found in Table 13.
For the baseline, we compare our methods with other start-of-the-art methods, including random walk-based methods : RWplus (Zhang & Zhang, 2010), sequence-based methods: AngularQA (Conover et al., 2019), and 3D structrue-based methods: VOroMQA (Olechnovivc & Venclovas, 2017), 3DCNN (Derevyanko et al., 2018).
The results of these baselines we reused from Baldassarre et al.
(2020) reports.
Another baseline HGNN (Feng et al., 2019) is reproduced by us with same training strategy as our methods.
Because our hypergraph based methods can jointly learn node and graph embeddings, the losses in equation 56 can be weighted as L total = µ L l +(1 − µ ) L g and co-optimized by Adam Optimizer with
L 2 regularization, where µ = 0 .
5 in our experiment.
We use grid search to select hyper-parameters and more details can be found in Table 8.
In addition, we use CASP10-12 for training and valuation, CASP 13 for testing to further evaluate the efﬁciency of our methods (the range of hyper-parameters can see Table 9), and the results are shown in Table 12.
I.5. P ROTEIN F OLD C LASSIFICATION
Datasets and settings The dataset that we used for training, validation and test is SCOPe 1.
data set of Hou et al.
This dataset includes 16,712 proteins covering 7 structural classes with total of 1195 folds.
The 3D structures of proteins are obtained from SCOpe 1.75 database (Murzin et al., 1995), in which each protein save in a PDB ﬁle.
The datasets have three test sets: 1) Fold, where proteins from the same superfamily do not appear in the training set; 2) Family, in which proteins from the same family are not present in the training set; 3) Family, where proteins from the same family that are present in the training set.
For baselines, we include sequence-based methods pre-trained unsupervised on millions of protein sequences:
Rao et al.
(2019); Bepler & Berger (2018)
; Strodthoff et al.
(2020), 3D structure based model: Kipf & Welling (2017); Diehl (2019); Baldassarre et al.
(2020) and Gligorijevic et al.
(2020), who process the sequence with LSTM ﬁrst and then apply GCNN.
The accuracy of the above baselines is reused from Hermosilla et al.
(2021) reported.
We adopt Adam optimizer to minimize the cross-entropy loss of our methods.
Hyper-parameters search range can see Table 7.
I. O VER -S MOOTHING A NALYSIS .
Table 14 shows that our proposed SHSC signiﬁcantly alleviates the performance descending with the increase of layers.
Given the same layers, it can be observed that our SHSC almost outperforms the other methods for all cases, especially at a deep layer, which demonstrates the beneﬁts of deep model and the long-range information around hypergraph.
Furthermore, It should be noted that the optimal layer numbers K of our SHSC are generally larger than other models, due to the only one linear layer avoids the over-ﬁtting problem.
These results also reveal that HyperGCN, HGNN and our GHCN all suffer from severe over-smoothing issue, limiting the power of neural network to capture high-order relationships. [Table]
I. S ENSITIVITY A NALYSIS
The proposed SHSC model performance on co-authorship Cora, co-citation cora and co-citation
Pubmed with different σ , β and α is reported in Figure 8.
For σ , we can see that the best choice will vary depending on the data set but mainly concentrated around − 0 .
[Table]
which verify that the effect of hyperedges degree is various and it has a negative effect in most cases.
For β , with the growth of β , the performance of SHSC stably increases
[Table]
which means that the diffusion kernel is useful for information aggregation.
Moreover, the fact that the performance remains stable on cora (coauthorship) and pubmed (cocitation) when β is at 0.8-1.0 suggests we can only adjust this hyper-parameter at a range of large value.
For α , the tendency of it is similar to β [Table] so we can also adjust this hyper-parameter at a range of large value to obtain a satisfying performance. [Table]
[Table]
[Figure]
I. T HE S PECTRUM A NALYSIS OF SHSC
We calculate the eigenvalues of (cid:16) β (cid:80) Kk
=1 α k K ˜ T k
+ (1 − β )
I (cid:17) with various ( α, β ) on NTU dataset ( |V| = 2012 )
and count the number of eigenvalues in different size ranges, which is shown in the Table 15.
The table suggests that our SHSC is able to capture both low and high frequency information of the graph signal, depending on the selection of the appropriate hyper-parameters ( α, β ) .
[Table]
I. A BLATION A NALYSIS
In order to verify the effectiveness of our proposed edge-dependent vertex weight Q and the necessity of re-normalization trick, we conduct an ablation analysis and report the results in Table 16.
Specially, w/o Q is a variant of our methods that replaces the edge-dependent vertex weight Q with the edge-independent vertex weight matrix H .
w/o renormalization represents the variants without re-normalization trick.
Form the reported results we can learn that both Q and renormalization are efﬁcient for hypergpraph learning, respectively.
Moreover, on Cora and Pubmed, the performance gap between w/o renormalization and with both GHCN reveals the advantage of renormalization on disconnected hypergraph dataset.
This phenomenon is mainly caused by the row in the adjacent matrix corresponding to an isolated point is 0 (Figure 4), resulting in a direct loss of its vertex information (Figure 4).
And the renormalization trick adding the self-loop to matrix K can maintain the features of isolated vertices during aggregation.
Compared with GHCN, the performance impact of renormalization on SHSC seems to be smaller.
This is because we add initial features to the information gathered by the neighbors, which reduces the information loss of isolated vertices
[Table]
[Table]
I. E XAMPLES OF ρ
We give many examples for ρ , including random , x − 1 , sigmoid ( x ) , 1 √ 2 σ exp ( − ( x − ¯ x ) 2 2 σ 2 ) , log ( x ) , exp ( x ) and exp ( − x ) , and the experimental results on citation networks can be seen the table 17 .
Regarding learnable ρ , we will do as a future work.
[Table]
I. R UNNING T IME AND C OMPUTATIONAL C OMPLEXITY
Firstly, we analyze the theoretical computational complexity of our GHCN and SHSC:
For GHCN, the computational cost is the O ( | E | d ) , where | E | is the total edge count in equivalent undigraph.
Each sparse matrix multiplication ˜ TX costs | E | d .
And the computational of HGNN is the same as GHCN.
For SHSC, the computational cost is the O
( K | E | d + K |V| d ) , which includes K sparse matrix multiplication and K summation over ﬁlters( |V| d is the cost of adding features X ).
Then, we compare the running time between our GHCN and SHSC with existing models in Table 18 and the results illustrate that our methods are of the same order of magnitude as SOTA’s approach UniGNN and outperform the HyperGCN and HGAT [Table] [Table]
I. T HE EDGE - DEPENDENT VERTEX WEIGHTS VISUALIZATION I.12. V ISUAL O BJECT .
I. A DDITIONAL E XPERIMENTS OF B ASELINES .
[Table]
[Table]