G RAPH N EURAL N ETWORKS
FOR
S OFT
S EMI -
S UPERVISED L EARNING ON H YPERGRAPHS
Anonymous authors
Paper under double-blind review
A BSTRACT
Graph-based semi-supervised learning (SSL) assigns labels to initially unlabelled vertices in a graph.
Graph neural networks (GNNs), esp.
graph convolutional networks (GCNs), inspired the current-state-of-the art models for graph-based
SSL problems.
GCNs inherently assume that the labels of interest are numerical or categorical variables.
However, in many real-world applications such as coauthorship networks, recommendation networks, etc., vertex labels can be naturally represented by probability distributions or histograms.
Moreover, real-world network datasets have complex relationships going beyond pairwise associations.
These relationships can be modelled naturally and ﬂexibly by hypergraphs.
In this paper, we explore GNNs for graph-based SSL of histograms.
Motivated by complex relationships (those going beyond pairwise) in real-world networks, we propose a novel method for directed hypergraphs.
Our work builds upon existing works on graph-based SSL of histograms derived from the theory of optimal transportation.
A key contribution of this paper is to establish generalisation error bounds for a one-layer GNN within the framework of algorithmic stability.
We also demonstrate our proposed methods’ effectiveness through detailed experimentation on real-world data.
We have made the code available.
I NTRODUCTION
In the last decade, deep learning models have been successfully embraced in many different ﬁelds and proved to achieve unprecedented performance on a vast range of applications Krizhevsky et al.
; Goodfellow et al.
; Bahdanau et al.
; LeCun et al.
Graph Convolutional
Network (GCN) Kipf & Welling (2017) was recently proposed as an adaptation of a particular deep learning model (i e., convolutional neural networks Lecun et al.
(1998)) to enable handling graph-structured data.
GCN was shown to be, in particular, effective in semi-supervised learning on attributed graphs.
GCNs have inspired the current state-of-the art models for graph-based SSL Wu et al. (2019a);
Veli ˇ ckovi ´ c et al.
; Vashishth et al.
GCNs inherently assume that the labels of interest are numerical or categorical variables.
However, in many real-world applications such as co-authorship networks, recommendation networks, etc., vertex labels can be naturally represented by probability distributions or histograms.
Moreover, these real-world network datasets have complex relationships going beyond pairwise associations.
Such relationships can be modelled naturally and ﬂexibly by hypergraphs.
Moreover, hypergraphs can encode additional relationships with directions as illustrated in Figure 1 and these hypergraphs are directed hypergraphs Gallo et al.
Inspired by a prior work Solomon et al.
(2014) that generalised label propagation to graph-based soft
SSL setting and motivated by the fact that GNNs have inspired state-of-the-art models for traditional graph-based SSL, we make the following contributions
• We explore GNNs for soft SSL in which vertex labels are probability distributions.
Motivated by real-world applications, we propose DHN (Directed Hypergraph Network), a novel method for directed hypergraphs.
DHN can be applied for soft-SSL using existing tools from optimal transportation (Section 3).
• We provide generalisation error bounds for a one-layer GNN within the framework of algorithmic stability.
We establish that such models, which use ﬁlters with bounded eigenvalues
independent of graph size, can satisfy the strong notion of uniform stability and thus is generalisable.
In particular, the algorithmic stability of a one-layer GNN depends on the largest absolute eigenvalue of the graph convolution ﬁlter (Section 4).
• We demonstrate DHN’s effectiveness through detailed experimentation on real-world data.
In particular, we demonstrate superiority over state-of-the-art hypergraph-based neural networks.
We provide new empirical benchmarks for soft-SSL on directed hypergraphs and make the code available to foster reproducible research (Section 5).
R ELATED WORK
Graph-based deep learning: Geometric deep learning Bronstein et al.
(2017) is an umbrella phrase for emerging techniques attempting to generalise (structured) deep neural networks to nonEuclidean domains such as graphs and manifolds.
GCN Kipf & Welling (2017) and their various extensions are the current state-of-the art for graph-based SSL Wu et al. (2019a); Veli ˇ ckovi ´ c et al.
; Monti et al.
; Vashishth et al.
; Ma et al.
; Qu et al.
and
graphbased unsupervised learning Hamilton et al.
(2017a) Veli ˇ ckovi ´ c et al.
(2019) problems.
GCNs have also been applied for semi-supervised graph classiﬁcation Li et al.
The reader is referred to a comprehensive literature review Bronstein et al.
(2017) and extensive surveys Hamilton et al.
(2017b)
; Battaglia et al.
; Zhang et al.
; Wu et al.
(2019b)
; Sun et al.
; Zhou et al.
(2018) on this topic.
Recently, graph-based deep models (also messagepassing neural networks Gilmer et al.
(2017)) have been analysed theoretically Dehmamy et al.
; Ying et al.
; Knyazev et al.
; Chen et al.
(2019b)
; Maron et al.
; Morris et al.
; Xu et al. (2019b); Chen et al.
(2019a)
; Kawamoto (2018); Xu et al.
; Chen et al.
We note a work on stability and generalisation bounds of GCNs Verma & Zhang (2019).
Learning on hypergraphs: Hypergraph is a combinatorial structure consisting of vertices and hyperedges, where each hyperedge is allowed to connect any number of vertices, thus generalizing graphs.
This additional ﬂexibility facilitates the capture of higher order interactions among objects; applications have been found in many ﬁelds such as computer vision Govindu (2005), network clustering Demir et al.
(2008), folksonomies Ghoshal et al.
, cellular networks Klamt et al.
(2009), and community detection Chien et al.
The seminal work on hypergraphs Zhou et al.
(2007) introduced the popular Agarwal et al.
Feng et al.
(2018; 2019) clique expansion of a hypergraph.
Hypergraph neural networks (HGNN)
Feng et al.
(2019) use the clique expansion while HyperGCN Yadati et al.
(2019) uses the mediator- based Laplacian to extend GCNs to hypergraphs.
Another line of work uses the mathematically appealing tensor methods Shashua et al.
; Bul`o & Pelillo (2009); Kolda & Bader (2009) but they are limited to uniform hypergraphs.
Recent developments work for arbitrary hypergraphs and fully exploit the hypergraph structure Hein et al.
; Zhang et al.
; Chan & Liang (2018);
Wendler et al.
These developments are motivated from the spectral theory of hypergraphs which is an active area of research Li & Milenkovic (2017);
Chien et al.
; Chitra & Raphael (2019); Li & Milenkovic (2018b); Li et al.
(2018a) ; Li & Milenkovic (2018a) Graph-based soft SSL: Researchers have shown that using unlabelled data during training can improve label prediction signiﬁcantly Chapelle et al.
; Zhu et al.
; Subramanya & Talukdar (2014)
; Yang et al.
While most methods assume that labels of interest are numerical or categorical variables, other works “soften” this assumption and handle “soft labels” such as histograms Corduneanu & Jaakkola (2005); Tsuda (2005).
One way of propagating histograms is to minimise the Kullback-Leibler (KL) divergence Subramanya & Bilmes (2011).
Recent studies have replaced the metric-agnostic KL divergence with metric-aware Wasserstein distance (interactions between histogram bins) for graphs Solomon et al.
and hypergraphs Gao et al.
Embeddings in Wasserstein space: There exist at least a couple of recent works that embed
Gaussian distributions in the Wasserstein space Muzellec & Cuturi (2018); Zhu et al.
Inspired by a recent work Frogner et al.
(2019), in this work, we focus on embedding input data as a discrete probability distrirbution on a ﬁxed support set.
The Wasserstein distance and its gradient require the solution of a linear program Villani (2008) and are costly to compute Peyr ´ e & Cuturi (2019).
A popular efﬁcient approximation is the Sinkhorn divergence Cuturi (2013) in which the underlying problem is regularised and is computed efﬁciently by a ﬁxed-point iteration.
Recent works have shown that it is suitable for gradient-based optimisation through automatic differentiation Frogner et al.
; Genevay et al.
; Frogner et al.
A couple of recent works compute Wasserstein distance between graph pairs Xu et al.
(2019a)
; Vayer et al.
M ETHOD
In this section, we ﬁrst describe soft SSL on directed hypergraphs and then propose DHN (Directed
Hypergraph Network) for the problem.
3. D IRECTED HYPERGRAPH
A directed hypergraph Gallo et al.
(1993) is an ordered pair H
=
( V, E d ) where V = { v 1 , · · · , v n } is a set of n vertices and
[Equation]
is a set of m directed hyperedges.
Each element in E d is an ordered pair ( t, h )
where t ⊆
V is the tail and
h ⊆ V is the head with t (cid:54) = ∅ , h (cid:54) = ∅ .
Denote the set of all undirected hyperedges by E .
E = (cid:91) ( t,
[Equation]
E d (cid:16) t ∪ h (cid:17) .
Denote I ∈ { 0 , 1 } | V |×| E | to be the incidence matrix of E i.e. I ( v, e )
=
if v ∈ e and 0 otherwise.
3. S OFT SSL ON DIRECTED HYPERGRAPHS
We consider the problem of predicting probability distributions for the vertices in H =
( V, E d ) given a typically small subset V k ⊆ V of vertices with known distributions.
In this work, we are concerned with discrete distributions modelled on a metric space i.e. an ordered pair ( M, C ) in which M is a set and C is the cost function (metric) associated with the set.
Furthermore, we assume that we are provided with a feature matrix, X V ∈ R n × D V , in which each vertex v ∈ V is represented by a D V -dimensional feature vector x v (here n = | V | ).
We are also provided with a hyperedge feature matrix
X E ∈ R m × D E with x e , e ∈ E as D E -dimensional feature representations (here m = | E d | ).
Our objective is to learn a labelling function Z = h (cid:0) H , X V , X E (cid:1) that maps each vertex to a probability distribution in the space of discrete probability distributions P F ( M ) on F atoms ( F is number of histogram bins) deﬁned on the metric space ( M, C ) .
The cost function C can be represented by a non-negative symmetric matrix of size F × F .
Note that each row of Z ∈ [0 , 1] n × F maps each vertex v ∈ V to a probability distribution Z v ∈ [0 , 1] F .
The function h is going to be trained on a supervised loss, L ,w.r.t to the vertices in V k so that the trained h can be used to predict distributions of all the vertices in V \ V k .
We now give an example application and then the details of the labelling function h followed by the supervised loss L .
Example application: Predicting topic distributions of authors in co-authorship networks can be posed as a soft SSL problem on directed hypergraphs.
V represents the set of authors, E the set of all collaborations (documents), E d the citation relationships among the documents, F the number of possible research interests of authors (Machine Learning, Theoretical Computer Science, etc.), X V and X E any available features on the authors and documents respectively (e.g. text attributes).
3. DHN (D IRECTED H YPERGRAPH N ETWORK )
Hypergraphs contain hyperedges in which relationships can go beyond pairwise and hence are challenging to deal with.
A ﬂexible way to embed vertices of a hypergraph is to “approximate” the hypergraph by a suitable graph and then apply traditional graph-based methods on the vertices.
Two notable candidates of h are Hypergraph neural network (HGNN)
Feng et al.
(2019) and Hypergraph
Convolutional Network (HyperGCN) Yadati et al.
HGNN uses the clique expansion of the hypergraph Zhou et al.
(2007) while HyperGCN uses the mediator-based Laplacian Chan & Liang (2018) to approximate the input hypergraph.
However, they are restricted to undirected hyperedges and also cannot exploit the hyperedge feature matrix X E .
A key idea of our approach is to treat each hyperedge e ∈ E as a vertex of the graph G =
( E, E d ) .
We then pass G through a graph neural network to obtain H E = f GNN ( G , X E ) so that the initial features, X E , are reﬁned to H E .
We then propose the layer-wise propagation rule of DHN as:
[Equation]
where [ · , · ] denotes concatenation, t is the time step, I is the incidence matrix, H ( t +1)
E
= σ 1 (cid:16) I T H ( t ) V (cid:17) for t = 1 , · · · , τ − 1 , H (0) E =
f GNN ( G , X E ) , σ and σ 1 are non-linear activation functions, and τ is the total number of time steps with H (0)
V = X V .
Note that the labelling function
Z = h ( H , X V , X E ) = softmax (cid:16) H τV (cid:17) where softmax is applied row-wise.
3. T HE SUPERVISED LOSS L
A crucial observation here is that because of the softmax layer, the output of h is (already) inherently a probability distribution.
For each vertex v ∈ V k , the predicted distribution Z v and the (known) true distribution Y v must be “close” to each other.
A natural way to compare probability distributions is to use the KL-divergence between Y v and Z v .
However, KL-divergence cannot exploit the metric space ( M, C ) and suffers from stability issues Chen et al.
In this work, we use the more stable Wasserstein distance to exploit the metric space Gao et al.
[Equation]
where W p [Equation] [Equation] =
[Equation]
For discrete distributions, W p is the solution of a linear program.
For practical purposes, we compute the regularised distance using the Sinkhorn algorithm.
Please see Appendix A.1 for more details.
Optimisation: We call DHN optimised with the Wasserstein loss as Soft-DHN.
All parameters are learned using stochastic gradient descent (SGD).
Please see Appendix for time complexity.
T HEORETICAL A NALYSIS : G ENERALISATION ERROR BOUND
In this section, we establish generalisation error bounds for a one-layer GNN by extending the results of a traditional GCN Verma & Zhang (2019) to the soft SSL setting with Wasserstein loss.
The main novelty is to generalise the error bounds to the learning problem “valued in the Wasserstein space”.
The main challenge is that the Wasserstein space is an abstract metric space without linear structure.
The section is organised as follows.
We ﬁrst introduce all the notations needed (ego-graph view, semi-supervised learning setting, etc.).
We then give single layer and SGD bounds using the nota- tions.
We ﬁnally give the main result (Theorem 1) which states that a GNN trained with Wasserstein loss has the same generalisation error bound as the traditional GCN (trained with cross entropy).
Let G =
( V, E ) be a connected graph with | V | = n vertices.
We consider GNN of a single layer
[Equation]
where X ∈ R n × d is the feature matrix ( n is the number of vertices in a graph, d is the dimension of the feature vectors),
K = g ( L G ) is a graph ﬁlter (typically symmetrically normalised adjacency with self loops, and L G ∈ R n × n is the graph Laplacian), and Θ ∈ R d × F is the set of parameters.
We note that our proposed DHN falls under this formulation in special circumstances.
Speciﬁcally if the non-linearity σ 1 in Equation 1 is removed we get the kernel K = II T (also known as the clique expansion of the hypergraph Zhou et al.
The non-linearity σ in Equation 3 is the softmax function acting on each row of the product g
( L G ) X Θ ∈ R n × F ; the output is of dimension n × F , where each output row is a discrete probability distribution, i.e., [Equation] and f
[Equation]
X
[Equation] Θ [Equation] 1 F = 1 n where 1 F = (1 , . .
. , 1) F ∈ R F , and similarly for 1 n .
Without loss of generality, we assume d = 1 .
Note that in order for the output to be nontrivial probability distributions, we must assume F > 1 .
We adopt an ego-graph view (Verma & Zhang, 2019, formula (2)) to simplify our discussion for local behavior of the soft GCN at a particular vertex.
Whenever no confusion arises, we identify a vertices x and χ in the graph G with their respective D -dimensional feature vectors.
Thus the output of f at x ∈ V is
[Equation]
where N ( x ) denotes for the one-hop neighborhood of x with respect to the adjacency relation deﬁned by matrix K , and K xχ ∈ R stands for the entry in K ∈ R n × n that describes the adjacency relation between vertices x and χ .
Let E x := (cid:80) χ ∈N ( x ) K xχ χ ∈ R so that f ( x, Θ) =
σ ( E x · Θ) .
We consider the supervised learning setting, and learn GNN from the training set { z i =
( x i , y i ) , i = 1 , . . .
, m } sampled i.i.d.
from the product space V × P F with respect to probability distribution D on this product space, where P F is the space of discrete probability distributions on F atoms.
The output of softmax lies in P F , which is a convex cone.
For any new data z = ( x, y ) ∼ D , we evaluate the performance of GNN f using a Wasserstein cost
[Equation]
Here the Wasserstein cost is deﬁned with respect to a cost function penalizing moving masses across bins.
Since we are working only with histograms in GNN, we shall use a cost function C ∈ R F × F that is deﬁned for pairs of histogram bins.
The transport problem is a linear program with z = f ( x, Θ) as in Equation 10.
Please see Appendix Sections A.3.1 and A.3.3 for the assumptions made and the notations used for Algorithmic stability.
We now derive single layer and SGD bounds.
4.0. S INGLE L AYER B OUND
By the triangle inequality,
[Equation]
By (Villani, 2008, Theorem 6.13), recall that the diameter of the support is D , we have
[Equation]
where (cid:107)·(cid:107) TV is the total variation distance, which by deﬁnition is
[Equation]
where (cid:107)·(cid:107) 1 is the L 1 -distance on R F .
Since the softmax function is Lipschitz continuous, we have
[Equation]
and thus
[Equation]
[Equation]
where we used notation g λ :
= sup x ∈ V | E x | as deﬁned in Verma & Zhang (2019), which is known to be upper bounded by λ max G , the spectrum of the graph Laplacian L G with largest absolute value.
4.0. SGD B OUND
It now remains to bound E A (cid:107) Θ S − Θ S (cid:48) (cid:107) 1 resulting from the SGD iterations.
The main technical challenge, as noted before, is to generalise the results to the Wasserstein space which is an abstract metric space without linear structure.
Speciﬁcally, we have to modify the “gradient” in the Wasserstein space as the straightforward version Verma & Zhang (2019) does not satisfy the Lipschitz condition required in the algorithmic stability framework.
To the best of our knowledge this modiﬁcation is not seen in existing literature and can be thought of as a generalisation of the “gradient clipping” operation Hardt et al.
The entire proof is in the Appendix A.3.4.
We state the main result here.
[Equation]
Combining equation 4 and equation 23 gives us
[Equation]
Therefore, we actually have the uniform algorithmic stability equation 14 holds with
[Equation]
Note that here β m =
O ( 1 m ) (needed to obtain a tight generalisation bound).
4. P UTTING E VERYTHING T OGETHER
Lemma 1 (Verma & Zhang, 2019, Lemma 4): Let λ maxG be the maximum absolute eigenvalue of
L G .
Let G x be the ego-graph of a vertex x ∈ V with corresponding maximum absolute eigenvalue λ
maxG x .
Then the following eigenvalue (singular value) bound holds ∀ x ∈ V ,
[Equation]
Lemma 2 (Verma & Zhang, 2019, Theorem 2)
A uniform stable randomised algorithm ( A S , β m ) with a bounded loss function 0 ≤ (cid:96) ( A S , y ) ≤ B , satisﬁes the following generalisation bound with probability at least 1 − δ , over the random draw of S, y with δ ∈ (0 , 1) ,
[Equation]
where R ( A S ) is the generalisation error/risk and R emp ( A S ) is the empirical error.
Please see Appendix A.3.2 for deﬁnitions.
Finally, by Equation 8, and Equation 9, our result i.e. Equationimmediately gives the following theorem:
Theorem 1 Let A S be a one-layer GNN algorithm (of 3) equipped with the graph
convolutional ﬁlter g
( L G ) and trained on a dataset S for T iterations.
Let the loss and activation functions be Lipschitz-continuous and smooth.
Then the following expected generalisation gap holds with probability at least 1 − δ , δ ∈ { 0 , 1 } :
[Equation]
where the expectation E SGD is taken over the randomness inherent in SGD, m is the no. training samples, and B is a constant which depends on the loss function.
Our theorem states that GNN trained withe Wasserstein loss enjoys the same generalisation error bound of the traditional GCN (trained with cross entropy).
We now discuss experiments.
[Table]
To demonstrate the effectiveness of our proposed DHN, we conducted experiments on 5 real-world directed hypergraphs.
Four of them are Co-authorship datasets and one of them is a recommendation dataset.
Table 1 shows the statistics of the datasets.
For more details on the construction of the datasets, please see Appendix A.6.
5. E XPERIMENTAL SETUP
Inspired by the experimental setups of prior related works Kipf & Welling (2017)
; Liao et al.
(2019), we tune hyperparameters using the Cora citation network dataset alone and use the optimal hyperparameters for all the other datasets.
We hyperparameterise the cost matrix (base metric of the
Wasserstein distance) as follows:
[Equation] [Equation]
 
The cost matrix C is an F × F matrix ( F is the number of histogram bins) with ones on the diagonal and a hyperparameter η elsewhere.
We could have used a matrix of all η s.
But it is no different from a matrix of all ones from the optimisation perspective
and so we used the above more general matrix.
Details of hyperparameter tuning and optimal hyperparameters are in Appendix A.5.
5. B ASELINES
We used both Wasserstein distance and KL divergence to train different models.
As already noted, we used the Sinkhorn algorithm to compute the (regularised) Wasserstein distance.
Please see Appendix A.1 for more details.
We compared DHN with the following baselines:
• KL-MLP : We used a simple multi-layer perceptron (MLP) on the features of the vertices and trained it using KL-divergence • OT-MLP :
We trained another MLP with the Wasserstein distance as the loss function.
Note that this baseline and the previous baseline do not use the structure (graph / hypergraph) • KLR-MLP : We regularised an MLP with explicit KL-divergence-based regularisation that uses the structure (graph / hypergraph) Subramanya & Bilmes (2011).
• OTR-MLP :
We regularised an MLP with explicit Wasserestein-distance-based regularisation that uses the structure (graph / hypergraph)
Solomon et al.
For hypergraphs we used the clique expansion of the hypergraph Gao et al.
• KL-HGNN / KL-HyperGCN : We trained the different GCN-based methods on hypergraphs with KL divergence loss function on the labelled vertices.
• Soft-HGNN / Soft-HyperGCN : We trained the different GCN-based methods on hypergraphs with the Wasserstein distance as the loss function.
Metric for comparison: We use the mean squared error (MSE) between true and predicted distributions on the test set of vertices.
Table 2 shows MSEs on the test split for all the three datasets. [Table]
[Table]
D ISCUSSION
We used a simple one-layer architecture for our proposed DHN and a 2 -hop simpliﬁed GCN Wu et al. (2019a) as the GNN model on the graph G =
( E, E d ) i.e.
[Equation]
where A is the symmetically normalised adjacency (with self loops) of the graph G .
We demonstrate that this simple model is effective enough through an ablation study in Appendix Table 4.
Our results demonstrate strong performances across all the datasets esp.
on the co-authorships networks.
Speciﬁcally, we observe that Soft models (that use the Wasserstein loss) are almost always superior to their counterparts that use the KL divergence as the loss function.
This is because the Soft models can exploit the distance matrix C while KL-divergence does not.
Moreover, our proposed DHN outperforms two strong hypergraph baselines viz.
HGNN Feng et al.
(2019) and HyperGCN Yadati et al.
We believe this is because they do not exploit the rich structural information in the directed hyperedges (connections among hyperedges) while our proposed DHN does exploit them.
We also experimented on standard graph benchmark node-classiﬁcation datasets such as Cora, Citeseer, and Pubmed by treating the class label as one-hot probability distribution.
We used the Soft variants of GCN Kipf & Welling (2017), Simple GCN Wu et al. (2019a), and GAT Veliˇckovi´c et al.
We achieved competitive results as shown in Appendix Table
C ONCLUSION
We have proposed DHN, a novel method for soft SSL on directed hypergraphs.
DHN can effectively propagate histograms to unknown vertices by integrating vertex features, directed hyperedges and undirected hypergraph structure.
As a key contribution, we have established generalisation bounds for DHN within the framework of algorithmic stability.
We have also demonstrated DHN’s effectiveness through detailed experimentation on real-world hypergraph datasets.
A A PPENDIX
The appendix is organised as follows:
Regularised Wasserstein and the Sinkhorn algorithm 2.
Time complexity of the proposed DHN 3.
Proofs and more notations used for theoretical analysis 4.
Additional experiments (Graph-based soft SSL and Ablation Study) 5.
Details of hyperparameters 6. Sources of real-world datasets A. R EGULARISED W ASSERSTEIN AND THE S INKHORN A LGORITHM
For discrete distributions, W p of Equation 2 is the solution of a linear program: W
[Equation]
where for an arbitrary element q ∈ P F , [ q ] i stands for the probability mass in the i th bin.
A.1. S INKHORN DIVERGENCE
The expensive linear program 10 can be efﬁciently solved by entropic regularisation Cuturi (2013):
[Equation]
where log( · ) is applied element-wise and λ ≥ 0 is a hyperparameter.
The optimal solution π ∗ for λ > 0 takes the following form.
[Equation]
where diag( z ) is a diagonal matrix with the components of z in the diagonal places.
Sinkhorn algorithm Cuturi (2013): We optimise Equation 11 for r and c via matrix balancing, i.e., start with an initial K :=
exp( − C p λ ) and alternately ensure the marginal constraints are satisﬁed until convergence: r ← z./ ( Kc ) c
← y./ ( K T r ) where ./ is element-wise division.
We use the above efﬁcient algorithm in our experiments.
A. T IME COMPLEXITY OF DHN
We consider the problem of predicting probability distributions for the vertices in H =
( V, E d ) given a typically small subset V k ⊆ V of vertices with known distributions.
In this work, we are concerned with discrete distributions modelled on a complete separable metric space ( M, C ) .
Furthermore, we assume that we are provided with a feature matrix, X V ∈ R n × D V , in which each vertex v ∈ V is represented by a D v -dimensional feature vector x v (here n = | V | ).
We are also provided with a hyperedge feature matrix
X E ∈ R m × D E with x e , e ∈ E as D E -dimensional feature representations.
Let T be the time taken for the Sinkhorn algorithm on all the vertices with known distributions.
Note that T can be approximated in near-linear time Altschuler et al.
Further, let T be the total number of epochs of training.
Deﬁne
[Equation]
The time complexity oa one-layer DHN is O (cid:16) | E d |· D E · h 1+ | E |· D V · h 2 (cid:17) where h 1 is the number of hidden units of the GNN layer and h 2 is the number of output channels.
A. T HEORETICAL A NALYSIS : P ROOFS
This section is organised as follows:
Assumptions/notations 2.
Deﬁnitions of generalisation and empirical errors 3.
Framework of Algorithmic Stability 4.
Proof of SGD Bound A. A SSUMPTIONS /N OTATIONS
To avoid unnecessary technical complications, assume the histogram admits a geometric realisation over the one-dimensional Euclidean space, such that the i th bin is placed at location b i ∈ R , and set
[Equation]
Without loss of generality, we assume b 1 ≤ b 2 ≤ · · · ≤ b F , and write h i :=
b i +1 − b i ≥ 0 for all i = 1 , . . .
, F − 1 .
Denote the diameter of the support by D := max 1 ≤ i,j ≤ F | b i − b
j | =
b F − b 1 .
We take the Wasserstein cost as the Wasserstein- 1 distance: W ( µ, ν ) :
= W 1 ( µ, ν ) .
In this particular one-dimensional setting, we have a particularly simple form for the cost function:
[Equation]
[Equation] where F µ : R → [0 , 1] , F ν : R → [0 , 1] are the cumulative distribution functions of µ , ν , respectively; F − 1 µ , F − 1 ν are the generalized inverses of F µ and F ν , respectively, deﬁned as (similar for
F − 1 ν )
[Equation]
This characterisation is seen in any standard literature on optimal transport, e.g., (Villani, 2003,
Theorem 2.18).
A.3. D EFINITIONS : G ENERALISATION AND E MPIRICAL E RRROS
Let the learning algorithm
A S on a dataset S be a function from ζ m to ( Y ) X .
where X is the input
Hilber space, Y is the output Hilbert space, ζ = X × Y .
The training set of datapoints, labels is
S = { z 1 = ( x 1 , y 1 ) , · · · , z k = ( x k , y k )) } .
Let the loss function be (cid:96) : ζ m × ζ → R .
Then the generalisation error or risk R ( A S ) is deined as
[Equation]
where p ( z ) is the probability of seeing the sample z ∈ S .
The empirical error, on the other hand, is deﬁned as
[Equation]
A.3. A LGORITHMIC S TABILITY
Denote S ⊂ V × P F for an arbitrary training data set sampled with respect to distribution D .
A learning algorithm for GNN, denoted as A , maps a training set S to a trained GNN f ( · , Θ S ) :
x (cid:55)→ f ( x, Θ S ) .
Let S (cid:48) be another training data set that differs from S by exactly one data.
Our goal in this section is to establish a uniform stability
[Equation]
[Equation]
with β m =
O (1 /m ) , which in turn can be used to establish the generalisation error bound of the form (Verma & Zhang, 2019, Theorem 1), following the framework Hardt et al.
for SGDbased learning algorithms.
Following the arguments in Verma & Zhang (2019) [Equation] this boils down to checking a few Lipschitz properties for the cost function [Equation]
A.3. SGD B OUND : P ROOF
Note that
[Equation]
and from the single-layer bound
[Equation]
Uniform stability:
[Equation]
It now remains to bound E A (cid:107) Θ S − Θ S (cid:48) (cid:107) 1 resulting from the SGD iterations.
Given training set S , applying SGD to GNN amounts to performing the updates
[Equation]
where η > 0 is the learning rate and z i t
= ( x i t , y i t ) are random data i.i.d.
uniformly sampled from the training set.
By the simple formulae equation 15 for one-dimensional optimal transport, we can explicitly write out for any parameter set Θ and data z
= ( x, y )
[Equation]
where again we used notation [ y ] i to denote the probability mass of y ∈ P F in the i th bin, for all i = 1 , . .
, F .
Thus
[Equation]
where sgn {·} is the sign function [Equation] and δ ik is the Kronecker delta notation.
The second equality used the speciﬁc form of the derivative of the softmax function.
Unfortunately, this gradient is not
Lipschitz continuous due to the sign function in the second term.
Nevertheless, if we use a “modiﬁed gradient” that drops the second term, i
[Equation]
e., choose to update the parameter Θ k in the direction
[Equation] [Equation]
then obviously this new choice of “descent” direction is certainly Lipschitz continuous, as
[Equation]
Therefore, if we deﬁne
[Equation]
then the stochastic update algorithm
[Equation]
[Equation]
In fact, equation 21 is exactly the GNN analogy of (Verma & Zhang, 2019, Lemma 1) which establishes the stability for the “same sample loss” case.
The “different sample loss” analogy, or the lemma in (Verma & Zhang, 2019, Lemma 2), can be trivially obtained by the deﬁnition equation [Equation]
In fact, noting that
[Equation]
we easily obtain
[Equation]
and it follows that (cid:13)
[Equation]
[Equation]
Putting together equation 21 and equation
[Equation]
we obtain the following analogy of (Verma & Zhang, 2019, Lemma 3): Starting with two training data sets S , S (cid:48) that differs by exactly one sample, after each iteration t we have
[Equation]
Solving this ﬁrst-order recursion gives the stability after T random update steps:
[Equation]
Combining equation 16 and equation 23 gives us
[Equation]
Therefore, we actually have the uniform algorithmic stability equation 17 holds with
[Equation]
A. A DDITIONAL E XPERIMENTS
Dataset
GCN
Soft
[Table]
GCN
GAT
Soft
[Table]
Cora
[Table] Citeseer [Table] A.4. A BLATION S TUDY
In this section, we compared our proposed one-layer DHN against deeper layers and without exploiting directed hyperedges.
[Table]
A. D ETAILS OF HYPERPARAMETERS
Inspired by the experimental setups of prior related works Kipf & Welling (2017)
; Liao et al.
(2019), we tune hyperparameters using the Cora co-authorship network dataset alone.
The optimal hyperparameters are ﬁxed and then used for all the other datasets.
Table 5 shows the list of hyperparameters used in the datasets.
Prior works Kipf & Welling (2017);
Liao et al.
(2019) have extensively performed tuning of hyperparameters such as hidden size, learning rate, etc and we ﬁxed their reported optimal hyperparameters.
It should be noted that self training and co-training methods Li et al.
(2018b) can also be used in case of absence of validation data.
We hyperparameterise the cost matrix (base metric of the Wasserstein distance) as follows:
[Equation]

Table 6 shows the best results on the validation split of Cora (with optimal hyperaparameters).
The training set had 140 vertices, the validation set 1000 vertices and the rest of the vertices were used to test the models.
The results reported are after 200 epochs of training with a seed value of
[Table]
A. S OURCES OF THE REAL - WORLD DATASETS
Co-authorship data : All authors co-authoring a paper are in one hyperedge.
We used the author data 2 to get the co-authorship hypergraph for cora.
We manually constructed the DBLP dataset from
Arnetminer 3 .
[Table]
A.6. C ONSTRUCTION OF THE DBLP DATASET
We downloaded the entire dblp data from https://aminer.org/lab-datasets/ citation/DBLP-citation-Jan8.tar.bz Tang et al.
The steps for constructing the dblp dataset used in the paper are as follows:
• We deﬁned a set of 5 conference categories (histograms for the SSL task) as “algorithms”, “database”, “datamining”, “intelligence”, and “vision” • For a total of 4304 venues in the entire dblp dataset we took papers from only a subset of venues from https://en.wikipedia.org/wiki/List_of_computer_ science_conferences corresponding to the above 5 conferences • From the venues of the above 5 conference categories, we got 22535 authors publishing at least two documents for a total of• We took the abstracts of all these 43413 documents, constructed a dictionary of the most frequent words (words with frequency more than 100 ) and this gave us a dictionary size of • We then extracted the 117215 citation links among these documents
A.6. C ONSTRUCTION OF THE A MAZON O FFICE P RODUCT DATASET
We downloaded the entire Amazon data He & McAuley (2016); McAuley et al. (2015).
The steps for constructing the dataset used in the paper are as follows:
• We downloaded the ofﬁce product ratings subset from the entire dataset • We constructed a hypergraph of items with each hyperedge representing a user connecting all the items that they bought • We removed hyperedges of size• We connected a pair of hyperedges (bi-directional) if they had more than 1 item in common A.6. C ONSTRUCTION OF THE ACM DATASET We downloaded the entire ACM data from https://lfs.aminer.org/lab-datasets/ citation/acm.v9.zip Tang et al.
The steps for curating the dataset in the paper are as follows:
• Based on the number of papers published, we identiﬁed the six most popular venues: “Journal of Computational Physics”, “IEEE Transactions on Pattern Analysis and Machine Intelligence”, “Automatica (Journal of IFAC)”, “IEEE Transactions on Information Theory”, “Expert Systems with Apllications: An International Journal”, and “IEEE Transactions on Computers” •
We then listed the set of all authors published in these venues (we got a total ofauthors).
• We ﬁnally obtained the citation relationships of all the documents co-authored by these authors (total number of documents is 25511 and total number of citations is 59884 )
A.6. D ETAILS OF THE A R X IV DATASET
We downloaded the entire arXiv dataset Clement et al.
from https://github.com/ mattbierbaum/arxiv-public-datasets/releases/tag/v0.2.0 .
The steps for curating the dataset in the paper are as follows:
• We removed papers without any authors and got a total of
edges • We extracted 67 , 28 , 683 citation edges among these papers •
The total number of authors in these papers is 7 , 90 ,