N EURAL S TRUCTURED P REDICTION FOR I NDUCTIVE N ODE C LASSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper studies node classiﬁcation in the inductive setting, i e., aiming to learn a model on labeled training graphs and generalize it to infer node labels on unlabeled test graphs.
This problem has been extensively studied with graph neural networks (GNNs) by learning effective node representations, as well as traditional structured prediction methods for modeling the structured output of node labels, e g., conditional random ﬁelds (CRFs).
In this paper, we present a new approach called the Structured Markov Network (SMN), which combines the advantages of both worlds.
SMN deﬁnes ﬂexible potential functions of CRFs with GNNs.
However, learning such a model is nontrivial as it involves optimizing a maximin game with high-cost inference.
Inspired by the underlying connection between the joint and marginal distributions deﬁned by Markov networks, we propose to solve an approximate version of the optimization problem as a proxy, which yields a near-optimal solution, making learning more efﬁcient.
Extensive experiments on two settings show our approach outperforms many competitive baselines.
I NTRODUCTION
Graph-structured data (e.g., World Wide Web) are ubiquitous in the real world, covering a variety of applications.
This paper studies node classiﬁcation, a fundamental problem in graph machine learning.
Most existing efforts of node classiﬁcation focus on the transductive setting (Kipf & Welling, 2017;
Veliˇckovi´c et al., 2018), i.e., using a small set of labeled nodes in a graph to classify the rest of nodes.
In this paper, we study node classiﬁcation in the inductive setting (Hamilton et al., 2017), which is receiving growing interest in the graph machine learning community.
Given a set of training graphs with all the nodes labeled, we aim to learn a classiﬁer to infer the nodes in unlabeled test graphs.
This problem has been recently studied with graph neural networks (GNNs) (Kipf & Welling, 2017;
Hamilton et al., 2017; Gilmer et al., 2017;
Veli ˇ ckovi ´ c et al., 2018).
GNNs infer the marginal label distribution of each node by learning useful node representations based on node features and edges.
Speciﬁcally, the representation of each node is iteratively updated by the representations of its neighbors.
Once a GNN is learned on training graphs, it can be further applied to test graphs to infer node labels.
Owing to the high capacity of nonlinear neural architectures, GNNs achieve impressive results on many datasets.
However, one limitation of GNNs is that they ignore the joint dependency of node labels, and thus node labels are predicted separately without modeling structured output.
Indeed, modeling structured output has been widely explored by the literature of structured prediction (BakIr et al., 2007).
Structured prediction methods predict node labels collectively, so the label prediction of each node can be improved according to the predicted labels of neighboring nodes.
One representative approach is the conditional random ﬁeld (CRF) (Lafferty et al., 2001).
A CRF models the joint distribution of node labels with Markov networks, and thus training CRFs becomes a learning task in graphical models, while predicting node labels corresponds to an inference task.
Typically, the potential functions in CRFs are parameterized as log-linear functions, which suffer from low model capacities.
One remedy for this is to deﬁne potential functions with GNNs (Ma et al., 2018;
Qu et al., 2019).
However, most of the effective methods for learning CRFs involve a maximin game (Wainwright & Jordan, 2008; Sutton & McCallum, 2012), making learning often hard to converge, especially when GNNs are used to parameterize potential functions.
Besides, as learning
CRFs requires doing inference on the graphical models, the combined model requires a long run time.
In this paper, we address these challenges by proposing SMN (Structured Markov Network), which is high in capacity, efﬁcient in learning, and able to model the joint dependency of node labels.
SMN is inspired by theoretical works in graphical models (Wainwright & Jordan, 2008), which reveal close connections between the joint label distribution and the node/edge marginal label distribution in a
Markov network.
Based on that, we approximate the original optimization problem with a proxy problem, where the potential functions in CRFs are deﬁned by combining a collection of node/edge pseudomarginal distributions, which are parameterized by GNNs that satisfy a few simple constraints.
This proxy problem can be easily solved by maximizing the data likelihood on each node and edge, which yields a near-optimal joint label distribution on training graphs.
Once the model is learned, we apply it to test graphs and run loopy belief propagation (Murphy et al., 1999) to infer node labels.
Experiments on two settings against both GNNs and CRFs prove the effectiveness of our approach.
R ELATED W ORK
Graph neural networks (GNNs) perform node classiﬁcation by learning useful node representations.
The idea is to iteratively update the representation of each node according to its own features as well as its neighboring nodes.
Well-known models include graph convolutional networks (Kipf & Welling, 2017), graph attention networks (Veli ˇ ckovi ´ c et al., 2018), and message passing neural networks (Gilmer et al., 2017).
Most earlier efforts focus on designing GNNs for node classiﬁcation in the transductive setting (Yang et al., 2016; Kipf & Welling, 2017; Gao & Ji, 2019; Xhonneux et al., 2020), and many recent works move to study the inductive setting (Hamilton et al., 2017; Gao et al., 2018; Chiang et al., 2019;
Li et al., 2019; Chen et al., 2020a; Zeng et al., 2020).
Because of high model capacity and efﬁcient end-to-end training, GNNs achieve impressive results on inductive node classiﬁcation.
Despite the great success, GNNs only try to model the marginal distribution of each node label and predict node labels separately without considering their joint dependency.
In contrast, SMN models joint distributions of node labels by combining GNNs with CRFs, which can thus predicts node labels collectively to improve the result.
See Sec.
4.3 for the detailed discussion.
Another type of approach for inductive node classiﬁcation is structured prediction, which focuses on modeling the dependency of node labels, so that the predicted node labels are more consistent.
One representative approach is structured SVM (Tsochantaridis et al., 2005; Finley & Joachims, 2008;
Sarawagi & Gupta, 2008), but it lacks a probabilistic interpretation to handle the uncertainty of the prediction.
Another representative probabilistic approach is conditional random ﬁeld (Lafferty et al., 2001; Sutton & McCallum, 2006), which models the distribution of output spaces by using a Markov network.
CRFs have been proven effective in many applications, such as POS tagging (Lafferty et al., 2001), shallow parsing (Sha & Pereira, 2003), image labeling (He et al., 2004), and sequence labeling (Lample et al., 2016; Ma & Hovy, 2016;
Liu et al., 2018).
Nevertheless, the potential functions in CRFs are typically deﬁned as log-linear functions, suffering from low model capacity.
There are also some recent works trying to combine GNNs and CRFs.
Some works use GNNs to solve inference problems in graphical models (Dai et al., 2016;
Satorras et al., 2019; Zhang et al., 2020;
Chen et al., 2020b; Satorras & Welling, 2020).
In contrast, our approach uses GNNs to parameterize the potential functions in CRFs, which is in a similar vein to Ma et al. ; Qu et al. Ma et al. ; Wang et al.
Among them, Ma et al.
(2018) and Qu et al.
(2019) optimize the pseudolikelihood (Besag, 1975) for model learning, and Wang et al. (2021) optimizes a cross-entropy loss on each single node, which can yield poor approximation of the true joint likelihood (Koller & Friedman, 2009; Sutton & McCallum, 2012).
Our approach instead solves a proxy problem, which yields a near-optimal solution to the original problem of maximizing likelihood, and thus achieves superior results. For Ma et al.
(2019), they model joint dependency of node labels by designing a generative model, which is evaluated in the transductive setting. Different from them, our approach designs a discriminative model based on CRF, which has weaker assumptions on the underlying generative process, and we focus on the inductive setting. For Ma et al. (2021), they study continuous node labels, which is different from our setting of node classiﬁcation with discrete labels.
Lastly, learning CRFs has been widely explored in graphical models.
Many methods perform learning by solving a maximin game as a surrogate for data likelihood maximization (Sutton & McCallum, 2009; 2012; Wiseman & Kim, 2019).
Nevertheless, these maximin games are often hard to optimize.
To avoid solving the maximin game, we follow Wainwright et al.
and build an approximate optimization problem as a proxy, which is much easier to solve and yields a near-optimal solution.
P RELIMINARY 3. P ROBLEM D EFINITION
This paper focuses on inductive node classiﬁcation (Hamilton et al., 2017), where we are given some labeled training graphs and unlabeled test graphs.
Each training graph is represented as ( y ∗ V , x V , E ) , where x V and y ∗ V are features and labels of a set of nodes V , and E is a set of edges.
For each test graph ( x ˜ V , ˜ E ) , only node features x ˜ V and edges ˜
E are observed.
Based on that, we aim to solve:
• Learning.
On training graphs, learning a probabilistic model to approximate p ( y V | x V , E ) .
• Inference.
For each test graph, inferring node labels y ∗ ˜ V according to the distribution p ( y ˜ V | x ˜ V , ˜ E ) .
The problem can be solved by GNNs and CRFs.
Next, we introduce the details of these methods.
3. G RAPH N EURAL N ETWORKS
For inductive node classiﬁcation, graph neural networks (GNNs) learn node representations to predict marginal label distributions of nodes.
GNNs assume all node labels are independent conditioned on node features and edges, so the joint label distribution is factorized into a set of marginals as below:
[Equation]
Each marginal distribution p θ ( y s | x V , E ) is modeled as a categorical distribution over label candidates, and the label probabilities are computed by applying a linear softmax classiﬁer to the representation of node s .
In general, node representations are learned via the message passing mechanism (Gilmer et al., 2017), which brings high capacity to GNNs.
Also, owing to the factorization in Eq.
(1), learning and inference can be easily solved in GNNs, where we simply need to compute loss and make prediction on each node separately.
However, GNNs approximate only the marginal label distributions of nodes on training graphs, which may generalize badly and result in poor approximation of node marginal label distributions on test graphs.
Also, the labels of different nodes are separately predicted according to their own marginal label distributions, yet the joint dependency of node labels is ignored.
3. C ONDITIONAL R ANDOM F IELDS
In the context of the paper, conditional random ﬁelds (CRFs) build graphical models for node classiﬁcation.
A popular model is the pair-wise CRF, which formalizes the joint label distribution as:
[Equation]
where Z θ ( x V , E ) is the partition function.
θ s ( y s , x V , E ) and θ st ( y s , y t , x V , E ) are scalar scores contributed by each node s and each edge ( s, t ) .
In practice, these θ -functions can be either deﬁned as simple linear functions or complicated GNNs.
To make the notation concise, we will omit x V and
E in the θ -functions, e.g., simplifying θ s ( y s , x V , E ) as θ s ( y s ) .
With these θ -functions, CRFs are able to model the joint dependency of node labels and therefore achieve structured prediction.
However, learning CRFs to maximize likelihood p θ ( y ∗ V | x V , E ) on training graphs is nontrivial in general, as the partition function Z θ ( x V , E ) is typically intractable in graphs with loops.
Thus, a major line of research instead optimizes a maximin game equivalent to likelihood maximization (Wainwright & Jordan, 2008).
The maximin game for each training graph ( y ∗ V , x V , E ) is formalized as follows:
[Equation]
Here, q ( y V ) is a variational distribution on node labels, q s ( y s ) and q st ( y s , y t ) are its marginal distributions on nodes and edges.
H [ q ( y V )] := − E q ( y V )
[log q ( y V )] is the entropy of q ( y V ) .
Given the maximin game, q and θ can be alternatively optimized via coordinate descent (Sutton & McCallum, 2012).
In each iteration, we ﬁrst update the node and edge marginals { q s ( y s ) } s ∈ V , { q st ( y s , y t ) } ( s,
t ) ∈
E
Graph
NodeGNN
EdgeGNN s t u v τu τv τs τt
Pseudomarginals by node and edge GNNs ys yt yu yv
CRF for joint label distribution s t st u tu vs v uv s t τst t u τtu u v τuv v s τvs towards deﬁned by p θ .
This can be done by MCMC, but high, so inference is often used, such as loopy belief propagation (Murphy et al., 1999).
After q is optimized, we further update θ -functions with the node and edge marginals deﬁned by q via gradient descent.
The optimal θ -functions are characterized by the following moment-matching conditions:
[Equation]
where I a { b } is an indicator function whose value is 1 if a = b and 0 otherwise.
See Sec.
A and Sec.
B in appendix for detailed derivation of the maximin game as well as the moment-matching conditions.
Once the θ -functions are learned, they can be further applied to each test graph ( x ˜ V , ˜ E ) to predict the joint label distribution as p θ ( y ˜ V | x ˜ V , ˜ E ) .
Then the best label assignment y ∗
˜ V can be inferred by using approximate inference algorithms, such as loopy belief propagation (Murphy et al., 1999).
The major challenge of CRFs lies in learning.
On the one hand, learning relies on inference, meaning that we have to update { q s ( y s ) } s ∈ V , { q st ( y s , y t ) } ( s,t )
∈ E to approximate the node and edge marginals of p θ at each step, which can be expensive.
On the other hand, as learning involves a maximin game and the optimal q of the inner minimization problem in Eq.
(3) is intractable, we can only maximize an upper bound of the likelihood function for θ , making learning unstable.
The problem becomes even more severe when θ is parameterized by highly nonlinear neural models, e.g. GNNs.
M ODEL
Next, we introduce SMN.
SMN combines GNNs and CRFs by parameterizing potential functions in
CRFs with GNNs, and thus enjoys high capacity and can model the joint dependency of node labels.
However, as elaborated in Sec. 3.3, learning such a model on training graphs is challenging due to the maximin game in optimization.
Inspired by the connection between the joint and marginal distributions of CRFs, we instead construct a new optimization problem, which serves as a proxy for model learning.
Compared with the original maximin game, the proxy problem is much easier to solve, where we can simply train two GNNs to approximate the marginal label distributions on nodes and edges, and further combine these pseudomarginals (deﬁned in Prop. 1) into a near-optimal joint label distribution.
This joint label distribution can be further reﬁned by optimizing the maximin game, although it is optional and often unnecessary, as this distribution is often close enough to the optimal one.
With this proxy problem for model learning, learning becomes more stable and efﬁcient.
Afterwards, the learned model is used to predict the joint label distribution on test graphs.
Then we run loopy belief propagation to infer node labels.
Now, we introduce the details of our approach.
4. L EARNING
The learning task aims at training θ to maximize the log-likelihood function log p θ ( y ∗ V | x V , E ) for each training graph ( y ∗ V , x V , E ) , which is highly challenging.
Therefore, instead of directly optimizing this goal, we solve an approximate version of the problem as a proxy, which is training a node GNN and an edge GNN to maximize the log-likelihood of observed labels on nodes and edges.
The Proxy Problem.
The proxy problem is inspired by Wainwright & Jordan (2008), which points out that the marginal label distributions on nodes and edges deﬁned by a Markov network have inherent connections with the joint distribution.
This connection is stated in the proposition below.
Proposition 1 Consider a set of nonzero pseudomarginals { τ s ( y s ) } s ∈ V and { τ st ( y s , y t ) }
( st ) ∈ E which satisfy (cid:80)
y s τ st
( y s , y t ) =
τ t ( y t ) and (cid:80) y t
τ st ( y s , y t ) =
τ s ( y s ) for all ( s, t ) ∈ E .
If we parameterize the θ -functions of p θ in Eq.
(2) in the following way:
[Equation]
then { τ s ( y s ) } s ∈ V and { τ st ( y s , y t ) } ( s,t
) ∈
E are speciﬁed by a ﬁxed point of the sum-product loopy belief propagation algorithm when applied to the joint distribution p θ , which implies that:
[Equation]
The proof is provided in Sec.
C. With the proposition, we observe that if we parameterize the θfunctions by combining a set of pseudomarginals { τ s ( y s ) } s ∈ V and { τ st ( y s , y t ) } ( s,t ) ∈ E in the way deﬁned by Eq.
(5), then those pseudomarginals can well approximate the true marginals of the joint distribution p θ , i.e., τ s ( y s )
≈ p θ ( y s ) and τ st ( y s , y t )
≈ p θ ( y s , y t ) for all nodes s and edges ( s, t ) .
Given this precondition, if we further have τ s ( y s )
≈ I
y ∗ s { y s } and τ st ( y s , y t )
≈ I
( y ∗ s ,y ∗ t ) { ( y s , y t ) } , then the moment-matching conditions in Eq.
(4) for the optimal θ -functions are roughly satisﬁed.
This implies the joint distribution p θ ( y V | x V , E ) derived in this way is a near-optimal one.
With the observation, rather than directly using GNNs to parameterize the θ -functions, we use a node
GNN and an edge GNN to parameterize the pseudomarginals { τ s ( y s ) } s ∈ V and { τ st ( y s , y t ) } ( s,t ) ∈ E .
For the pseudomarginal τ s ( y s ) on node s , we apply the node GNN to node features x V and edges E , yielding a representation u s for node s .
Then we apply a softmax classiﬁer to u s to compute τ s ( y s ) :
[Equation]
where f maps a node representation to a |Y| -dimensional logit and Y is the node label set.
Similarly, we apply the edge GNN to compute a representation v s for each node s , and model τ st ( y s , y t ) as:
[Equation]
where g is a function mapping a pair of representations to a ( |Y| × |Y| ) -dimensional logit.
Given the parameterization, we construct the following problem as a proxy for learning θ -functions:
[Equation]
y s , y t )
[Equation]
[Equation] st ( y s , y t
[Equation]
, subject to θ s = log τ s ( y s )
[Equation]
and (cid:88)
[Equation]
for all nodes and edges, where d can be any divergence measure between two distributions.
By solving the above problem, { τ s ( y s ) } s ∈ V and { τ st ( y s , y t ) } ( s,
t ) ∈
E will be valid pseudomarginals which can well approximate the true labels, i.e., τ s ( y s )
≈ I
y ∗ s { y s } and τ st ( y s , y t )
≈ I
( y ∗ s ,y ∗ t ) { ( y s , y t ) } .
Then according to the constraint in the second line of Eq.
(9), θ -functions are formed in a way to enable τ s ( y s )
≈ p θ ( y s ) and τ st ( y s , y t )
≈ p θ ( y s , y t ) as stated in the Prop. 1.
Combining these two sets of formula results in p θ ( y s )
≈ I
y ∗ s { y s } and p θ ( y s , y t )
≈ I
y ∗ s { y s } .
We see that the moment-matching conditions in Eq.
(4) for the optimal joint label distribution are roughly achieved, implying that the derived joint distribution p θ ( y V | x V , E ) is a near-optimal solution to the original learning problem.
One good property of the proxy problem is that it can be solved easily.
The last consistency constraint (i.e. (cid:80) y s τ st ( y s , y t ) =
τ t ( y t ) and (cid:80) y t
τ st ( y s , y t ) =
τ s ( y s ) ) can be ignored during optimization, since by optimizing the objective function, the optimal pseudomarginals τ should well approximate the observed node and edge marginals, i.e., τ s ( y s )
≈ I
y ∗ s { y s } and τ st ( y s , y t )
≈ I
( y ∗ s ,y ∗ t ) { ( y s , y t ) } , and hence τ will almost naturally satisfy the consistency constraint.
We also tried some constrained optimization methods to handle the consistency constraint, but they yield no improvement.
See Sec.
D of appendix for more details.
Thus, we can simply train the pseudomarginals parameterized by GNNs to approximate the true node and edge labels on training graphs, i.e., minimizing d ( I y ∗ s { y s } , τ s ( y s )) and d ( I ( y ∗ s ,y ∗ t ) { ( y s , y t ) } , τ st ( y s , y t )) .
Then we build θ -functions as in Eq.
(5) to obtain a near-optimal joint distribution.
In practice, we choose d to be the KL divergence, yielding an objective for τ as:
[Equation]
This objective function is very intuitive, where we simply try to optimize the node GNN and edge
GNN to maximize the log-likelihood function of the observed labels on nodes and edges.
Reﬁnement.
By solving the proxy problem, we can obtain a near-optimal joint distribution.
In practice, we observe that when we have a large amount of training data, further reﬁning this joint distribution by solving the maximin game in Eq.
(3) for a few iterations can lead to further improvement.
Formally, each iteration of reﬁnement has two steps.
In the ﬁrst step, we run sum-product loopy belief propagation (Murphy et al., 1999), which yields a collection of node and edge marginals (i.e., { q s ( y s ) } s ∈ V and { q st ( y s , y t ) } ( s,t ) ∈ E ) as approximation to the marginals deﬁned by p θ .
In the second step, we update the θ -functions parameterized by the node and edge GNNs to maximize: (cid:88)
[Equation]
Intuitively, we treat the true label y ∗ s and ( y ∗ s , y ∗ t ) of each node and edge as positive examples, and encourage the θ -functions to raise up their scores.
Meanwhile, those labels sampled from q s ( y s ) and q st ( y s , y t ) act as negative examples, and the θ -functions are updated to decrease their scores.
4. I NFERENCE
After learning, we apply the node and edge GNNs to each test graph ( x ˜ V , ˜ E ) to compute the θfunctions, which are integrated into an approximate joint label distribution p θ ( y ˜ V | x ˜ V , ˜ E ) .
Then we use this distribution to infer the best label y ∗ ˜ s for each node ˜ s ∈ ˜ V , where two settings are considered.
Node-level Accuracy.
Typically, we care about the node-level accuracy, i.e., how likely we can correctly classify a node in test graphs.
Intuitively, the best label y ∗ ˜ s for each test node ˜ s
∈ ˜ V should be predicted as y ∗ ˜ s = arg max y ˜
s p
θ ( y ˜ s | x ˜ V , ˜ E ) , where p θ ( y ˜ s | x ˜ V , ˜ E ) is the marginal label distribution of node ˜ s induced by the joint p θ ( y ˜ V | x ˜ V , ˜ E ) .
In practice, the exact marginal is intractable, so we apply loopy belief propagation (Murphy et al., 1999) for approximate inference.
For each edge (˜ s, ˜ t ) in test graphs, we introduce a message function m ˜ t →
˜ s ( y ˜ s ) and iteratively update all messages as:
[Equation]
[Equation]
where N (˜ s ) denotes the set of neighboring nodes for node ˜ s .
Once the above process converges or after sufﬁcient iterations, the label of each node ˜ s can be inferred in the following way:
[Equation]
Graph-level Accuracy.
In some other cases, we might care about the graph-level accuracy, i.e., how likely we can correctly classify all nodes in a given test graph.
In this case, the best prediction of node labels is given by y ∗ ˜ V = arg max y ˜
V p
y ˜ V | x ˜ V , ˜ E ) .
This problem can be approximately solved by the max-product variant of loopy belief propagation, which simply replaces the sum over y ˜ t in Eq.
(12) with max (Weiss & Freeman, 2001).
Afterwards, the best node label can be still decoded via Eq.
4. D ISCUSSION
Next, we discuss the advantages of SMN over existing GNN methods.
(1) Compared with GNNs which only learn marginal label distributions on nodes, SMN approximates marginals on both nodes and edges.
These node and edge marginals mutually beneﬁt each other, allowing SMN to generalize better to test graphs.
(2) Besides, SMN tries to model the joint distribution of node labels, which allows for collective inference to improve node label prediction.
It also allows SMN to choose proper inference algorithms for different purposes, e.g., maximizing node-level or graph-level accuracy.
E XPERIMENT 5. D ATASETS
We consider datasets in two settings, which focus on node-level and graph-level accuracy respectively.
Node-level Accuracy.
The node-level accuracy measures how likely a model can predict the correct label of a node in test graphs .
We use the PPI dataset (Zitnik & Leskovec, 2017; Hamilton et al., 2017), which has 20 training graphs.
To make the dataset more challenging, we also try using only the ﬁrst 1/2/10 training graphs, yielding another three datasets PPI-1 , PPI-2 , and PPI-10 .
Besides, we also build a DBLP dataset from the citation network 1 in Tang et al.
Papers from eight conferences are treated as nodes, and we split them into three categories for classiﬁcation according to conference domains 2 .
For each paper, we compute the mean GloVe embedding 3 (Pennington et al., 2014) of words in the title and abstract as node features.
The training/validation/test graph is formed as the citation graph of papers published before 1999, from 2000 to 2009, after 2010 respectively.
Graph-level Accuracy.
The graph-level accuracy measures how likely a model can correctly classify all the nodes for a given test graph .
We construct three datasets from the Cora, Citeseer, and Pubmed datasets used for transductive node classiﬁcation (Yang et al., 2016).
Each raw dataset has a single graph.
For each training/validation/test node of the raw dataset, we treat its ego network 4 as a training/validation/test graph.
We denote the datasets as Cora* , Citeseer* , Pubmed* .
5. C OMPARED A LGORITHMS
Graph Neural Networks.
For GNNs, we choose a few well-known model architectures for comparison, including GCN (Kipf & Welling, 2017),
GraphSage (Hamilton et al., 2017), GAT (Veli ˇ ckovi ´ c et al., 2018), Graph U-Net (Gao & Ji, 2019) and GCNII (Chen et al., 2020a).
Conditional Random Fields.
For CRFs, we consider three variants.
(1) CRF-linear.
This variant uses linear θ -functions in Eq.
(2), which takes the features on nodes and edges for computation.
CRF-GNN.
This variant parameterizes the θ -functions as θ s ( y s ) =
f ( u s ) and θ st ( y s , y t ) =
g ( v s , v t ) , with f and g deﬁned in Eq.
(7) and Eq.
(8), where the node representations are generated by different
GNN architectures (e.g., CRF-GAT).
We train these models via the maximin game as in Eq.
(3) with sum-product loopy belief propagation.
(3) GMNN.
We also consider GMNN (Qu et al., 2019), an approach combining GNNs and CRFs, which optimizes the pseudolikelihood function for learning.
Our Approach.
For SMNs, we try different GNN architectures for deﬁning the node and edge GNNs (e.g., SMN-GAT).
By default, we only solve the proxy problem without performing reﬁnement.
We systematically compare the results with and without reﬁnement in part 2 of Sec. 5.5.
5. E VALUATION M ETRICS
On Cora*, Citeseer*, and Pubmed*, we report the percentage of test graphs where all the nodes are correctly classiﬁed (i.e., graph-level accuracy).
On DBLP and PPI, we report accuracy and micro-F based on the percentage of test nodes which are correctly classiﬁed (i.e., node-level accuracy).
For
Cora*, Citeseer*, and Pubmed*, we run each compared method with 10 different seeds to report the mean accuracy and the standard deviation.
For DBLP and PPI, we run each method with 5 seeds.
5. E XPERIMENTAL S ETUP
For GNNs, by default we use the same architectures (e.g., number of neurons, number of layers) as used in the original papers.
Adam (Kingma & Ba, 2015) is used for training.
For the edge GNN in Eq.
(8), we add a hyperparameter γ to control the annealing temperature of the logit g ( v s , v t ) before the softmax function during belief propagation.
Empirically, we ﬁnd that max-product belief propagation works better than the sum-product variant in most cases, so we use the max-product version by default.
By default, we do not run reﬁnement when training SMNs.
See Sec.
F for details.
[Table]
*, DBLP (in %).
SMNs achieve the best result.
[Table]
5. R ESULTS
Comparison with other methods.
The main results in the two settings are presented in Tab. 1 and
Tab. 2.
Compared against different GNN models, our approach achieves consistent improvement (the relative underperformance of SMN-GCN and SMN-SAGE is related to the capacity of the backbone
GNNs and is explained in Sec.
G.1)
by using these GNNs as backbone networks for approximating marginal label distributions on nodes and edges, which demonstrates SMNs are able to model the structured output of node labels by combining with CRFs, and thus achieve better results.
Besides, SMNs also achieve superior results to CRF-GNNs which are trained by directly solving the maximin game in Eq.
(3), as well as GMNN which optimizes the pseudolikelihood function.
This observation proves the advantage of our proposed proxy optimization problem for learning CRFs.
[Table] [Table]
Effect of reﬁnement.
By solving the proxy optimization problem in Eq.
(9), we can obtain a near-optimal joint label distribution on training graphs, based on which we may optionally reﬁne the distribution with the maximin game in Eq.
Next, we study the effect of reﬁnement, and we present the results in Tab. 4.
By only solving the proxy problem, our approach already achieves impressive results, showing that the proxy problem can well approximate the original learning problem.
Only on datasets with sufﬁcient labeled data (e.g., PPI
-10, PPI), reﬁnement leads to some improvement.
Model architecture.
SMN uses a node GNN and an edge GNN for computing node and edge marginals independently.
In practice, we can also use a shared GNN for both node and edge marginals.
We show results of this variant in Tab. 6, where it also achieves signiﬁcant improvement over GNNs.
Efﬁciency comparison.
We have seen SMNs achieve better classiﬁcation results than GNNs and CRFs.
Next, we further compare their efﬁciency by showing the run time on DBLP and PPI.
[Table] [Table]
For PPI, which has 121 labels, we only report the training times on a single label.
We use GAT as the backbone network for CRFs and SMNs.
GAT and CRF are trained for 1000 epochs to ensure convergence.
For the SMN, we train the node GNN and edge GNN for node/edge classiﬁcation as in Eq.
(10) with 1000 epochs.
The run times are presented in Tab. 3.
SMNs take twice as long for training than GAT, as a SMN needs to train a node GNN and an edge GNN.
Compared with CRFs,
SMNs are much more efﬁcient.
The reason is that the proxy optimization problem is much easier to solve, as it does not requires running expensive approximate inference algorithms.
[Figure] [Figure]
Comparison of learning methods.
Next, we investigate different methods for learning SMNs, including directly solving the maximin game, optimizing pseudolikelihood, and solving our proposed proxy problem.
We show the results for optimizing SMN-GAT in Tab. 5.
We see solving maximin game yields poor results due to unstable training.
Although the pseudolikelihood method performs much better, the result is still unsatisfactory as it is not a good approximation of the true likelihood.
By solving our proposed proxy problem, SMN achieves the best result, which proves its effectiveness.
Convergence analysis.
To better illustrate the advantage of the proxy problem for learning CRFs, we look into the training curves of SMNs, SMNs w/o proxy, and CRFs when optimizing the maximin game in Eq.
For SMNs, we optimize the node and edge GNNs on the proxy optimization problem in Eq.
(9) before doing reﬁnement with the maximin game, while for SMNs w/o proxy we directly perform reﬁnement with the maximin game without solving the proxy problem.
We show the results in Fig.
CRFs and SMNs w/o proxy suffer from high variance and low accuracy.
In contrast, owing to the near-optimal joint distribution found by solving the proxy problem, SMNs get much higher accuracy with lower variance even without reﬁnement (see initial results of SMNs at epoch 0).
Also, the reﬁnement process quickly converges after only a few epochs, showing good efﬁciency of SMNs.
Case study.
To intuitively see how SMNs outperform GNNs, we conduct some case studies on
Cora*.
We use GAT as backbone networks, and show the prediction made by the GAT (the node
GNN), the edge GNN, and SMN in Fig.
In all three cases shown in the ﬁgure, GAT (left column) makes inconsistent predictions on linked nodes, as it fails to model the structured output.
The edge
GNN (middle column) also makes a mistake in the bottom case.
Finally, by combining GAT and edge GNN with a CRF, the SMN (right column) is able to predict the correct labels for all nodes.
C ONCLUSION
This paper studied inductive node classiﬁcation, and we proposed SMN to combine GNNs and CRFs.
Inspired by the connection of joint and marginal distributions deﬁned by Markov networks, we designed a proxy optimization problem for efﬁcient model learning.
In the future, we plan to explore techniques on training GANs (Goodfellow et al., 2014) to better optimize the maximin game for learning.
In addition, SMNs model joint dependency of node labels by deﬁning potential functions on nodes and edges, and we also plan to further explore high-order local structures, e.g., triangles.
A D ERIVATION OF THE M AXIMIN G AME
As discussed in the Sec. 3, optimizing the joint label distribution p θ to maximize the log-likelihood log p θ ( y ∗ V | x V , E ) on a training graph ( y ∗ V , x V , E ) is equivalent to solving a maximin game.
In this section, we provide the detailed derivation.
Let ψ θ ( y V , x V , E ) be the potential function as below:
[Equation]
For each training graph ( y ∗ V , x V , E ) , we aim at maximizing the following log-likelihood function:
[Equation]
However, the term log (cid:80) y V
ψ
θ ( y V , x V , E ) is computationally intractable, as we need to sum over all the possible y V .
To solve the problem, we introduce a variational joint distribution q ( y V ) deﬁned on all node labels y V , and use the Jensen’s inequality to derive an estimation of the term log (cid:80)
y V ψ θ ( y V , x V , E ) as follows:
[Equation]
It is easy to see that the equation holds if and only if q ( y V )
=
p θ ( y V | x V , E ) , and hence:
[Equation]
By taking the above result into Eq.
[Equation]
we obtain:
[Equation]
As ψ
[Equation]
} , we have:
[Equation] [Equation]
with:
[Equation]
Therefore, optimizing θ to maximize the log-likelihood function is equivalent to solving the following maximin game:
[Equation]
B D ERIVATION OF THE M OMENT - MATCHING C ONDITIONS
In the CRF model deﬁned in the preliminary section, the parameter θ consists of the output values of all θ -functions.
In other words, θ = { θ s ( y s ) }
y s ∈Y ,s ∈ V ∪ { θ st ( y s , y t ) }
y s ∈Y ,y t ∈Y ,s ∈ V , where Y is the set of all the possible node labels.
By deﬁnition
[Equation]
p θ ( y V | x V , E ) belongs to the exponential family.
According to the property of exponential family distributions, log p θ ( y ∗ V | x V , E ) is strictly concave with respect to θ .
Therefore, the optimal θ is unique, which is characterized by the condition of ∂∂θ log p θ ( y ∗ V | x V , E )
= 0 .
Formally, ∂∂θ log p θ ( y ∗ V | x V , E ) can be computed as below:
[Equation]
[Equation]
For ∂∂θ log Z θ ( x V
[Equation]
E )
[Equation]
we have:
[Equation]
By combining the above two equations
[Equation]
we have:
[Equation]
The potential function ψ θ above is deﬁned as ψ θ ( y V , x V , E ) =
exp {
[Equation]
s ∈ V θ s (
[Equation]
s , x V , E ) + (cid:80) ( s,t ) ∈
E θ st ( y s , y t , x V , E ) } .
If we consider each speciﬁc scalar θ s (ˆ y s )
[Equation]
and taking the derivative with respect to the scalar to
[Equation]
we obtain:
[Equation]
which implies p θ (ˆ y s | x V , E )
=
I y ∗ s { ˆ y s } for the optimal θ .
Moreover, this equation holds for all s ∈ V and all ˆ y s ∈ Y .
Similarly, for each scalar θ st (ˆ y s , ˆ y t ) , we have that
[Equation]
is equivalent to p θ (ˆ y s , ˆ y t | x V , E ) =
I y ∗ s ,y ∗ t { ˆ y s , ˆ y t } .
This equation holds for all ( s, t ) ∈ E and all the choices of (ˆ y s , ˆ y t )
∈
Y × Y .
Therefore, the optimal θ -functions are characterized by the moment-matching conditions as below:
[Equation]
C P ROOF OF P ROPOSITION
Next, we prove Prop. 1.
We ﬁrst restate the proposition as follows:
Proposition For a node set V and an edge set E , consider a collection of nonzero pseudomarginals { τ s ( y s ) } s ∈ V and { τ st ( y s , y t ) }
( st ) ∈ E which satisfy (cid:80)
y s τ st
( y s , y t ) =
τ t ( y t ) and (cid:80) y t
τ st ( y s , y t ) =
τ s ( y s ) for all ( s, t ) ∈ E .
If we parameterize the θ -functions in p θ as follows:
[Equation]
then { τ s ( y s ) } s ∈ V and { τ st ( y s , y t ) } ( s,t
) ∈
E are speciﬁed by a ﬁxed point of the sum-product loopy belief propagation algorithm when applied to the joint distribution p θ , implying:
[Equation]
Proof: To prove the proposition, we ﬁrst summarize the workﬂow of the sum-product loopy belief propagation algorithm.
In sum-product loopy belief propagation, we introduce a message function m t → s ( y s ) for each edge ( s, t ) ∈ E .
Then we iteratively update all message functions as follows:
[Equation]
29) where N ( t ) represents the set of neighbors for node t .
Once the process converges or after sufﬁcient iterations, the approximation of the node marginals and the edge marginals (i.e., { q s ( y s ) } s ∈ V and { q st ( y s , y t ) } ( s,t ) ∈ E ) can be recovered by the message functions { m t → s ( y s ) } ( s,t ) ∈
E as follows:
[Equation]
Next, let us move back to our case, where we parameterize the θ -functions with a set of pseudomarginals as in Eq.
For such a speciﬁc parameterization of the θ -functions, we claim that one ﬁxed point of Eq.
(29) is achieved when m t → s ( y s )
= 1 for all ( s, t ) ∈ E .
To prove that, we notice that when all the message functions equal to 1, the left side of Eq.
(29) is apparently 1.
The right side of Eq.
(29) can be computed as below:
[Equation]
We can see that both the left side and the right side of Eq.
(29) are 1, and hence { m t → s ( y s )
= 1 }
( s,t ) ∈
E speciﬁes a ﬁxed point of sum-product loopy belief propagation.
For this ﬁxed point, q s ( y s ) can be computed as follows:
[Equation]
Similarly, we can compute q st ( y s , y t ) as:
[Equation]
From the above two equations, we can see that { τ s ( y s ) } s ∈ V and { τ st ( y s , y t ) }
( s,t ) ∈
E are speciﬁed by a ﬁxed point (i.e., m t → s ( y s )
= 1 for all ( s, t ) ∈ E ) of sum-product loopy belief propagation.
As sum-product loopy belief propagation often works well in practice to approximate the marginal distributions on nodes and edges, we thus have τ s ( y s )
≈ p θ ( y s ) for each node and τ st ( y s , y t )
≈ p θ ( y s , y t ) for each edge.
D S OLVING THE P ROXY P ROBLEM WITH C ONSTRAINED O PTIMIZATION
The key innovation of our proposed approach is on the proxy optimization problem which is used to approximate the original learning problem.
Formally, the proxy optimization problem is stated as:
[Equation]
I
[Equation]
, subject to θ s = log τ s ( y s ) ,
[Equation]
, and
[Equation]
) for all nodes and edges, where d can be any divergence measure between two distributions.
In our implementation, we ignore these consistency constraints, i.e., (cid:80) y s τ st ( y s , y t ) =
τ t ( y t ) and (cid:80) y t
τ st ( y s , y t ) =
τ s ( y s ) .
This ie because by by optimizing the objective, the obtained pseudomarginals τ would well approximate the observed node and edge marginals, i.e., τ s ( y s )
≈ I
y ∗ s { y s } and τ st ( y s , y t )
≈ I
( y ∗ s ,y ∗ t ) { ( y s , y t ) } , and hence τ would almost naturally satisfy the constraints.
To demonstrate ignoring the consistency constraint makes sense, we also tried a constrained optimization method for solving the proxy problem.
Speciﬁcally, we add a quadratic term to penalize the inconsistency between τ st ( y s , y t ) and τ s ( y s ) as well as τ t ( y t ) , resulting in the following problem:
[Equation]
,t ) ∈ E (cid:34) (cid:88)
y t { (cid:88)
y s τ st ( y s , y t ) − τ t ( y t ) } 2 + (cid:88)
y s { (cid:88)
y t τ st ( y s , y t )
− τ s ( y s ) }(cid:35) , subject to θ s = log τ s ( y s ) , θ st ( y s , y t ) =
log τ st ( y s , y t )
τ s ( y s )
τ t ( y t ) , (36) for all nodes and edges.
Again, d is a divergence measure between two distributions, and we choose to use the KL divergence.
α is a hyperparameter deciding the weight of the penalty term.
[Table]
We conduct empirical comparison of this constrained optimization method and our default implementation where the consistency constraint is ignored.
The results are presented in Tab. 7.
We can see that the constrained optimization method does not lead to improvement, which shows that ignoring the consistency constraint is empirically reasonable.
E U NDERSTANDING SMN S AS O PTIMIZING A S URROGATE FOR THE L OG - LIKELIHOOD F UNCTION
In the model section, we motivate SMNs from the moment-matching conditions of the optimal θfunctions.
Speciﬁcally, we initialize the θ -functions at a state where the moment-matching conditions are approximately satisﬁed, yielding a near-optimal joint distribution.
Then we further tune the θ -functions to solve the maximin game.
Besides this perspective, SMNs can also be understood as optimizing a surrogate for the log-likelihood function.
Next, we introduce the details.
Remember that maximizing the log-likelihood function is equivalent to solving a maximin game as:
[Equation]
L ( θ
[Equation]
q
[Equation]
H [ q ( y
[Equation] [Equation]
Here, q ( y V ) is a joint distribution on all the node labels.
q s ( y s ) and q st ( y s , y t ) are the corresponding marginal distributions.
Although the above maximin game is equivalent to the original problem of maximizing likelihood, solving the maximin game is nontrivial.
In particular, there are two key challenges, i.e., (1) how to specify constraints to characterize a valid joint distribution q ( y V ) and (2) how to compute its entropy
H ( q ) = − E q ( y V )
[log q ( y V )] .
To deal with the challenge, a common practice used in loopy belief propagation is to make the following two approximations: (1) Instead of specifying constraints to let q ( y V ) be a valid joint distribution, we introduce a set of pseudomarginals as approximation to a valid joint distribution.
Speciﬁcally, these pseudomarginals
are
denoted as
˜ q
= { q s
( y s
s ∈
V
∪ { q st ( y s , y t ) } ( s,t ) ∈ E , and they satisfy (cid:80) y s q st ( y s , y t ) =
q t ( y t ) and (cid:80) y t q st ( y s , y t ) =
q s ( y s ) for all ( s,
t ) ∈ E
We approximate
the entropy H ( q ) with
Bethe entropy approximation
H Bethe
˜ q ) ,
which
is
deﬁned as follows:
[Equation]
With the two approximations, we get the following maximin game as a surrogate for the likelihood maximization problem:
[Equation]
with:
[Equation]
This problem is known as the Bethe variational problem (BVP) (Wainwright & Jordan, 2008).
Such a problem can be solved by coordinate descent, where we alternate between updating ˜ q to minimize L Bethe ( θ, ˜ q ) and updating θ to maximize L Bethe ( θ, ˜ q ) .
According to Yedidia et al.
(2005), updating ˜ q to minimize L Bethe ( θ, ˜ q ) can be exactly achieved by running sum-product loopy belief propagation on p θ , where a ﬁxed point of the belief propagation algorithm yields a local optima of ˜ q .
On the other hand, updating θ to maximize L Bethe ( θ, ˜ q ) can be easily achieved by gradient ascent.
In addition to that, a stationary point ( θ ∗ , ˜ q ∗ ) of the above BVP is speciﬁed by following conditions:
[Equation]
According to Yedidia et al.
and Wainwright & Jordan (2008), the ﬁrst condition is equivalent to the condition that ˜ q ∗ is speciﬁed by a ﬁxed-point of sum-product loopy belief propagation.
The second condition states that the moment-matching conditions are satisﬁed, i.e., q s ( y s )
=
I y ∗ s { y s } on each node and q st ( y s , y t )
=
I ( y ∗ s ,y ∗ t ) { ( y s , y t ) } on each edge.
For our proposed approach SMN, it can be viewed as solving the BVP as deﬁned in Eq.
The pre-training stage of SMN aims at initializing θ at a state where the conditions of stationary points in
Eq.
(41) are approximately satisﬁed.
Then the ﬁne-tuning stage of SMN further adjusts θ to solve the maximin game by alternatively updating θ and ˜ q .
More speciﬁcally, in the pre-training stage, by initializing θ in the way deﬁned by Eq.
(27), the collection of pseudomarginal distributions { τ s ( y s ) }
s ∈ V and { τ st ( y s , y t ) } ( s,t
) ∈
E is speciﬁed by a ﬁxed point of sum-product loopy belief propagation according to proposition 3.1.
This implies that ∂∂ ˜ q L Bethe ( θ, ˜ q ) = 0 for ˜ q
= { τ s ( y s ) } s ∈ V ∪ { τ st ( y s , y t ) } ( s,t ) ∈ E .
Meanwhile, as { τ s } s ∈ V and { τ st } ( s,t ) ∈
E are learned to match the true labels y ∗ V on each training graph, we thus have τ s ( y s )
≈ I
y ∗ s { y s } on each node and τ st ( y s , y t )
≈ I
( y ∗ s ,y ∗ t ) { ( y s , y t ) } on each edge.
Therefore, the conditions in
Eq.
(41) are approximately satisﬁed by ( θ, ˜ q ) with ˜ q
= { τ s ( y s ) } s ∈ V ∪ { τ st ( y s , y t ) } ( s
,t ) ∈ E , which means that the pre-training stage yields a θ to roughly match the conditions of stationary points for the BVP in Eq.
After pre-training, the ﬁne-tuning stage of SMN is exactly trying to solve the maximin game of BVP in Eq.
(39), where the ﬁne-tuning stage alternates between updating ˜ q to minimize L Bethe ( θ, ˜ q ) via sum-product loopy belief propagation and updating θ to maximize
L Bethe ( θ, ˜ q ) via gradient ascent.
As a result [Figure] we see that the SMN can also be understood as solving the Bethe variational problem in
Eq.
(39), which acts as a surrogate for the log-likelihood function.
F E XPERIMENTAL D ETAILS
Next, we describe our experimental setup in more details.
F. D ATASET S TATISTICS
The statistics of the datasets used in our experiment are summarized in Tab. 8.
For the Cora, Citeseer,
Pubmed [Figure] and PPI datasets, they are under the MIT license.
[Table]
[Figure]
F. M ORE D ETAILS ABOUT THE DBLP D ATASET
Due to the limited space, the experiment section only brieﬂy describes the DBLP dataset
[Figure]
and we complement the details of the dataset here.
The DBLP dataset is constructed from the citation network 5 in Tang et al.
Scientiﬁc papers from eight conferences are treated as nodes, and we divide them into three categories for classiﬁcation based on conference domains 6 .
For each paper, we compute the mean GloVe embedding 7 (Pennington et al., 2014) of words in the title and abstract as node features.
We split the dataset into three disjoint graphs for training/validation/test.
The training graph contains papers published before(with 1999 included).
The validation graph contains papers published between 2000 and 2009 (with 2000 and 2009 included).
The test graph contains papers published after 2010 (with 2010 included).
There exists an undirected edge between two papers if one paper cites the other one.
Cross-split edges (e.g., an edge between a paper in the training set and a paper in the validation set) are removed.
F. A RCHITECTURE C HOICES
To facilitate reproducibility, we use the GNN module implementations of PyTorch Geometric (Fey & Lenssen, 2019), and follow the GNN models provided in the examples of the repository, unless otherwise mentioned.
Note that most architecture choices are not optimal on the benchmark datasets, but we did not tune them since we only need to show that our method brings consistent and signiﬁcant improvement.
GCN (Kipf & Welling, 2017).
We set the number of hidden neurons to 16, and the number of layers to 2.
ReLU (Nair & Hinton, 2010) is used as the activation function.
We do not dropout between GNN layers.
GraphSage (Hamilton et al., 2017).
We set the number of hidden neurons to 64, and the number of layers to 2.
ReLU (Nair & Hinton, 2010) is used as the activation function.
We do not dropout between GNN layers.
GAT (Veliˇckovi´c et al., 2018).
We set the number of hidden neurons to 256 per attention head, and the number of layers to 3.
The number of heads for each layer is set to 4, 4 and 6.
ELU (Clevert et al., 2016) is used as the activation function.
We do not dropout between GNN layers.
Graph U-Net (Gao & Ji, 2019).
We set the number of hidden neurons to 64 and the number of layers to 3.
We randomly dropout 20% of the edges from the adjacency matrix.
We do not dropout node features or between layers.
GCNII (Chen et al., 2020a).
We set the number of hidden neurons to 2048 for the citation datasets (Cora*, Citeseer*, Pubmed* and DBLP) and 256 for the PPI dataset.
We set the number of layers to 9.
ReLU (Nair & Hinton, 2010) is used as the activation function.
For PPI, layer normalization (Ba et al., 2016) is applied between the GCNII layers.
We do not dropout between GNN layers.
We set the strength α of the initial residual connection to 0.5, and the hyperparameter θ to compute the strength of the identity mapping to 1.
The g function.
In Eq.
(8) of the model section, we deﬁne g as a function mapping a pair of Ldimensional representations to a ( |Y| × |Y| ) -dimensional
logit.
Two variants of this function are used in our experiment.
For the PPI and DBLP dataset, we use the linear variant, where the pair of node representations are concatenated and plugged into a linear layer:
[Equation]
where W ∈ R ( |Y|×|Y| ) × 2 L is the weight matrix and b ∈ R |Y|×|Y| is the bias.
For the citation datasets (Cora*, Citeseer*, Pubmed*), we use the bilienar variant, where the pair of node representations are plugged in a bilinear mapping:
[Equation]
43) where W ∈ R |Y|× L is a weight matrix.
SMN with a shared GNN.
By default, the SMN uses a node GNN and an edge GNN to approximate the pseudomarginals on nodes and edges respectively.
In the experiment, we also consider using a shared GNN for both pseudomarginals on nodes and edges.
In other words, u s = v s , ∀ s ∈ V (see
Eq.
(7) and Eq.
All the other components are the same as the default SMN.
The results of this variant are shown in Tab.
6 of the experiment section.
F. H YPERPARAMETER C HOICES
GNNs and SMNs.
For the node classiﬁcation task, the learning rate of the node GNN τ s in GNNs and SMNs is presented in Tab. 9.
For the edge classiﬁcation task, the learning rate of the edge
GNN τ st is presented in Tab. 10.
For the temperature γ used in the edge GNN τ st of SMNs, we summarize its values in Tab. 11.
[Table]
[Table] [Table]
CRF-linear. For CRF-linear training, we set the learning rate to 5 × 10 − 4 . CRF-GNNs and SMN. For CRF and the reﬁnement stage of SMN, we set the learning rate to 1 × 10 − 5 .
GMNN.
For GMNN training, we set the learning rate to 5 × 10 − 3 .
F. C OMPUTATIONAL R ESOURCES
We run the experiment by using NVIDIA Tesla V100 GPUs with 16GB memory.
G A DDITIONAL R ESULTS
In this section, we present some additional experimental results.
G. A DDITIONAL A NALYSIS OF GNN A RCHITECTURES
In this analysis, we study the effect of node/edge GNN architectures on SMNs.
We ﬁx one of the
GNNs and change the capacity of the other (Fig. 4), then evaluate SMN-GAT on PPI-1-0 , a subset of PPI-1 that only contains its ﬁrst label.
The results show that our model beneﬁt from capacity gain in both node and edge GNNs, highlighting their effective synergy.
This also explains the underperformance of SMN-GCN in Tab. 1, where the edge GCN backbone with only two layers and 16 hidden neurons is incapable of modeling the edge label dependencies and thus drags the performance behind.
We also ﬁnd that the node and edge GNNs need not share the same backbone, and in many cases SMNs with different node and edge GNNs perform superior to those with same backbone (Fig. 4).
The expressiveness of edge GNNs is crucial to the performance of SMN.
Though we did not optimize the design of our edge GNNs, they have shown to be helpful in boosting the performance once plugged into our approach.
In the experiment, we report the graph-level accuracy on the Cora*, Citeseer*, and Pubmed* datasets, where SMNs consistently outperform other methods. Besides the graph-level accuracy, we also
   +HDGVLQWKH(GJH*1      + HDG V  L Q W KH  1 RGH  * 1 *$ *&1 6$*( (GJH*11$UFKLWHFWXUH * $  * &1  6$ * ( 1 RGH  * 11  $ U F K L W H F W X U H  
     
    compute the node-level accuracy on these datasets, and the results are reported in Tab.
We can see that our approach still consistently outperforms other methods in terms of node-level accuracy [Table] [Table]
G. C OMPARISON OF S UM - PRODUCT AND M AX - PRODUCT B ELIEF P ROPAGATION
As explained in section 4. [Table] the sum-product belief propagation algorithm is more applicable to the case of node-level accuracy, as it aims at inferring the marginal label distribution on each node.
Nevertheless, in practice we ﬁnd that the max-product algorithm usually achieves better empirical node-level accuracy.
For example, the results on the PPI-10 dataset are presented in Tab. 13.
[Table]
Because of the better empirical results
[Table]
we choose to use max-product belief propagation by default.
G. H YPERPARAMETER A NALYSIS
Finally
[Table]
We present analysis of the hyperparameter γ (i [Table] e., edge temperature) in Fig.
[Figure]