CGNF : C ONDITIONAL G RAPH N EURAL F IELDS
Anonymous authors
Paper under double-blind review
A BSTRACT
Graph convolutional networks have achieved tremendous success in the tasks of graph node classiﬁcation.
These models could learn a better node representation through encoding the graph structure and node features.
However, the correlation between the node labels are not considered.
In this paper, we propose a novel architecture for graph node classiﬁcation, named conditional graph neural ﬁelds ( CGNF ).
By integrating the conditional random ﬁelds (CRF) in the graph convolutional networks, we explicitly model a joint probability of the entire set of node labels, thus taking advantage of neighborhood label information in the node label prediction task.
Our model could have both the representation capacity of graph neural networks and the prediction power of CRFs.
Experiments on several graph datasets demonstrate effectiveness of CGNF .
I NTRODUCTION
Graph is an expressive representation for diverse datasets.
Node classiﬁcation on graphs has many real world applications.
For example, to classify publications we build citation networks where publications are nodes and the research ﬁelds are node labels (Hamilton et al., 2017).
To study the protein-disease relations we could construct protein-protein interaction (PPI) graphs where each protein is a node and diseases are treated as node labels.
Node classiﬁcation on graphs is a general task that creates knowledge and has real social impact, e.g., by exploring the similarities between proteins, we can learn novel protein target for treating diseases and use that knowledge to facilitate drug repurposing and safety monitoring (Zitnik et al., 2018; Ma et al., 2018).
Many earlier works take connectivity-based approaches to classify nodes, which mainly operate on the graph structure alone.
The underlying assumption is that nodes that are closely connected in the graph should have similar labels.
Among the most successful connectivity-based methods are node embedding techniques including (Perozzi et al., 2014; Grover & Leskovec, 2016a; Tang et al., 2015a; Hamilton et al., 2017; Kipf & Welling, 2016).
More recently, features of neighboring nodes are included in learning node embedding.
These feature-based approaches incorporate graph connectivities via regularization (Zhu et al., 2003; Belkin et al., 2006) or aggregation over local neighborhoods (Defferrard et al., 2016; Kipf & Welling, 2017; Chen et al., 2018a).
Among them, the graph convolutional networks (GCN) become one of the most successful methods of node embedding for arbitrary graphs with node features (Kipf & Welling, 2017; Defferrard et al., 2016;
Hamilton et al., 2017; Chen et al., 2018a).
Although GCN gets good node embeddings and node classiﬁcation performance via aggregating node features and graph connectivities, the correlation of node labels was not exploited.
In practice, the labels of neighborhood nodes in a graph are often mutually impacted.
For example, in citation networks the labels (ﬁelds of research) for neighborhood nodes (publications) are often consistent: a paper citing another tends to appear in similar or related ﬁelds to the cited one.
Unfortunately, we are not able to simply treat labels as additional node features because label information are not available in test set.
Instead, the key question is how to leverage label correlation in addition to node features in node classiﬁcation .
We notice that energy-based approaches such as the conditional random ﬁelds (CRF) (Lafferty et al., 2001) are desirable for capturing structured labels including correlation of node labels.
However, they may involve non-trivial optimization to perform predictions.
In this paper we propose a new deep architecture to extend the GCN by deﬁning an energy function to capture the dependencies between labels for better node embedding and classiﬁcation.
In particular, we model the pairwise label relation with a pairwise energy function and propose a new model integrating graph node embedding and conditional random ﬁelds.
The proposed model, coined conditional graph neural ﬁelds ( CGNF ), has the following contributions:
• CGNF is the ﬁrst work that bridges GCN and CRF to directly capture label correlations for more accurate node classiﬁcation in graphs.
Experimental results on several graph datasets demonstrated the advantage of CGNF over several baselines including GCN.
• We designed a speciﬁc graph-based energy function and proposed an efﬁcient way for approximate training as well as two efﬁcient inference schemes for predicting test labels.
• Experimental results on several graph datasets demonstrate our advantage over other graph neural networks such as GCN.
B ACKGROUND
Graph Convolutional Networks Over the past few years, several graph convolutional network models emerged to compute informative node feature representations for arbitrary graphs (Kipf &
Welling, 2017; Defferrard et al., 2016; Hamilton et al., 2017; Chen et al., 2018a).
GCN models learn node embeddings in the following manner:
Given each graph node initially attached with a feature vector, the embedding vector of each node are the transformed weighted sum of the feature vectors of its neighbors.
All nodes are simultaneously updated to perform a layer of forward propagation over neural networks.
The deeper the network, the larger the local neighborhood.
Thus global information is disseminated to each graph node for learning better node embeddings.
Speciﬁcally, given an undirected graph with input features matrix of nodes X and adjacency matrix of the underlying graph A , a multi-layer neural network is constructed on the graph with the following layer-wise propagation rule:
[Equation]
(1) where
˜ A =
A
+ I is the adjacency matrix with self-connections.
I is the identity matrix, D is a diagonal matrix such that ˜ D ii = (cid:80)
j ˜
A ij , W ( l ) is a layer-speciﬁc parameter matrix, H ( l ) is the node representation in the l th layer, H (0)
=
X , and σ is an activation function (e.g. ReLU or sigmoid).
However, labels and label correlation is not modeled in GCN.
Conditional Random Fields
The conditional random ﬁelds (CRF) is an undirected probabilistic graphical model commonly used for structured prediction tasks, e.g. named entity recognition and image segmentation.
Formally, given input feature x ∈ R d , CRF aims at ﬁnding the label set y that maximizes the conditional probability P ( y | x ) .
In particular, CRF deﬁnes a joint probability over cliques { x c } in the graph.
A clique is any fully connected subset of the the graph.
CRF represents
P ( y | x ) as a product of clique potential Φ c ( x c , y c ) .
Each potential function Φ c depends only on part of the feature input x denoted as x c and a label subset y a .
This factorization can allow us to represent P ( y | x ) much more efﬁciently (Sutton et al., 2012).
By making sure the factorization form of P ( y | x ) sum to 1, partition function Z ( x ) is used for normalization use as follows:
[Equation]
where Z ( x ) = (cid:80) y (cid:48) c (cid:81)
c
Φ a ( x c , y (cid:48) c ) is a normalizer required for a valid probability distribution.
CGNF M ETHOD 3. T ASK F ORMULATION
Consider the problem of multi-class node classiﬁcation.
The input graph G = { X , Y , A } includes node feature matrix X , adjacency matrix A , and the output node label matrix Y .
More speciﬁcally, we have X =
[ x 1 , x 2 , · · · , x N ] (cid:124) ∈ R N × D
and Y
=
[ y 1 , y 2 , · · · , y N ] (cid:124) ∈ R N × M , where N is the number of nodes in the graph, D is the dimension of node features, and M is the number of distinct labels.
The i th node has a feature vector x i , and a one-hot label vector
y i .
The graph connectivity is speciﬁed through the adjacency matrix A ∈ R N × N , where A i,j ∈ A indicates the edge weight
[Table]
between node i and node j .
We follow convention to denote ˆ A =
˜ D − 12 ˜
A ˜ D − 12 as a normalized version of ˜ A , which will be used in our model later.
We follow the traditional setting of GCN and formalize the learning in semi-supervised setting where we assume we observe the labels Y tr only for some of the nodes X tr .
Our goal is to predict the labels ˆ Y te for other unlabeled nodes X te .
To start with
[Table]
we consider { X tr
[Table]
Y tr
[Table]
A tr } as training set
[Table]
and { X te , Y te , A te } as test set.
3. T HE CGNF M ODEL
Given input graph G =
{ X
[Table]
Y , A } as previous described
[Table]
our goal is to predict node labels Y for graph G .
We can learn node embeddings and use them to make such predictions.
In this paper, we follow (Kipf & Welling, 2017) and use a two-layer GCN model given by Eq. 3.
[Equation]
Here W
[Equation]
∈
[Equation]
× M are weight parameters of GCN.
Softmax function is applied rowwise to make each row { h i } N 1 of GCN prediction H ∈ R N × M .
However, the training of GCN is conditionally independent for each node, and label correlation is ignored by this model.
To ﬁll this gap, we aim to integrate label correlations into the GCN model.
Energy based models, such as CRF, are ﬂexible ways to deﬁne the variable dependencies.
So in a general graph, considering both the impact of node features and label dependency, we can deﬁne the energy function as belows.
[Equation]
where c is a clique, E ∈ R M is the edge set, the unary potential ψ ( · ) is designed to measure the compatibility between observed node x i and its label y i ; and pairwise potential φ ( · ) is meant for capturing label correlation; γ is a mixture weight.
With this energy function, we can derive the
Gibbs distribution, i.e. the conditional probability P ( Y | X , A ) as given by Eq.
[Equation]
where Z = (cid:80) Y (cid:48) ∈Y exp( − E ( Y (cid:48) , X , A )) is the normalizer (or called partition function), E is the energy function which is an indicator of the likelihood of the corresponding relationships within the clique, with a higher energy conﬁguration having lower probability; Y is the label space consisting of unique label assignments Y (cid:48) .
Then our objective is to maximize this conditional probability.
To enhance Eq.
4 by the neural network representations of GCN, the energy function can be reformulated as:
[Equation]
where h i is the previously mentioned GCN prediction H on node
i and ˆ A
i,j is an element in the normalized adjacency matrix ˆ A .
Separating the energies for each node, we can further express the energy function as Eq.
[Equation]
where N ( i ) is the neighborhood of node i .
[Figure]
and second neighbor nodes of node i are utilized in two-layer GCN setting to generate graph prediction probability h i ∈
H using Eq.
3 and at the meantime we can get normalized graph adjacency matrix ˆ A .
Then we utilize speciﬁc designed graph-based energy function in Eq.consisting of unary and pairwise potential in Eq.
8 to make accurate node classiﬁcation using Eq.
Speciﬁcally, in a graph we deﬁne the unary potential for node i as the prediction loss of GCN, denoted as ψ ( y i , h i ) .
We also deﬁne the pairwise potential between node i and node j as φ ( y i , y j , ˆ A
i,j ) , which is impacted by two factors: the normalized edge weights ˆ A
i,j and label correlation weights U
y i ,y j .
The two potential functions are computed as Eq. 8.
[Equation]
where y i and y j are the corresponding label index of label vector y i and y j ; and
U y
i ,y j ∈ U is the learnable correlation/transition weight between label y i and y j .
Like in the traditional CRF, when all components for conditional probability in Eq.
5 are introduced throughly above, a common way is to use negative log likelihood as training objective function in
Eq.
9 to ﬁnd optimal parameters like W 0 , W 1 , U :
[Equation]
Using the energy based conditional probability, the inference of new labels is comparably easy in this model.
Notice that in Eq.
9 Since P ( Y | X , A ) ∝ exp( − E
( Y , X , A )) , after getting the parameters in the training phase, the inference of the test labels will simply be min Y E
( Y , X , A ) .
However, how to optimize the parameters by minimizing the negative log likelihood is a great challenge.
The computation of the normalizer Z ( X , A ) is often intractable, which causes a big problem in training.
In the next section, we will introduce two ways to solve the problem.
3. P ARAMETER O PTIMIZATION
In this section, we introduce the training ( parameter optimization) for CGNF .
It is implemented on the training data Y tr , X tr , A tr , but for simplicity here we omit the subscripts for all Y , X , A .
As we know, exact minimization of the energy based function Eq.
9 is intractable.
So we consider a simple but effective approximation method, the pseudo likelihood method.
A pseudo likelihood (Besag, 1975) provides a good approximation of the joint probability.
Use of the pseudo likelihood (PL) in place of the true likelihood function in a maximum likelihood analysis can lead to good estimates.
[Equation]
where N ( i ) are the neighbors of node x i .
The pseudo likelihood has the signiﬁcant advantage that it only requires normalizing over the possible labels at one node.
It could reduce the computation complexity of the normalizer from O ( M N ) to O ( MN ) .
Recall the conditional probability given by Eq 5 and Eq 7, for each node x i , the conditional probability given its neighborhood is calculated as follows:
[Equation]
y (cid:48)
i is all the possible label for node x i .
Notice here we do not have the 12 coefﬁcient on the pairwise potential, as we need to extract all dependent potentials of node x i .
The new training objective,
i
[Equation]
e. the negative log pseudo-log-likelihood, becomes the following form:
[Equation]
where i is the index of training nodes; ( · ) i,j is the element of a matrix at row i and column j ; and (cid:12) indicates element-wise multiplication.
3. I NFERENCE OF T EST L ABELS FOR CGNF
In the previous section, we introduced how to train the model and get the optimized parameter estimation.
As we explained in 3.2, after getting the model parameters, next the inference for new labels can be performed by optimizing Eq. 13.
[Equation]
where ˆ Y = concatenate ( Y tr , ˆ Y te ) .
To ﬁnd the minimum value we have to ﬁnd the “best” test labels, which is exponentially complex.
Hence we approximate the inference in two ways.
The simplest way is not to consider the correlation between test labels just use training labels to infer one test label each time.
[Equation]
Alternatively, we could use a dynamic-programming-style (DP) method to ﬁnd the minimum.
We randomly select a test node as the start and randomly permute the other test nodes.
Then we can do beam search along the order of the permuted test nodes (each beam search with beam size K can get
K best sets and we select just the best one).
Let us repeat this process T times, and then we compare all the T beam-search results and select the best one.
Then we can just select one best according to their energy.
The inference time in this case will be linear.
The algorithm is summarized below.
Input: Node feature matrix X , adjacency matrix A , times T and beam size K , energy function E ; Output: Estimated best output ˆ Y ∗ ; Initialize process times t ← 0 ; energy function score s ← inf ; candidate output Y c ← ∅ while t ≤ T do Reset beam B ← ∅ ;
Randomly permute test nodes { i } | Y tr |as { z i } | Y tr |where z
i is the new index for node i ; for i
= 1 to | Y te | do for all ˆ Y 1: i ∈ B do { ˆ Y 1: i } ← extend { ˆ Y 1: i − 1 } with y i ∈ label set Y ;
Append topK( { ˆ Y 1: i } ) using E into B ; end for
B ← topK ( B ) ; end for t ← t + 1 ;
Compute current minimum energy score s (cid:48)
← min ˆ Y ∈B E ( X , A , Y tr , ˆ Y ) ;
Compute current best output Y ∗ c ← arg min
ˆ Y ∈B E ( X , A , Y tr , ˆ Y ) ; if s (cid:48) ≤ s then s ← s (cid:48)
3. I NDUCTIVE L EARNING A CROSS G RAPHS
Following the framework of GCN, our node classiﬁcation task is essentially a semi-supervised learning task in one ﬁxed graph.
However, sometimes we have several graphs which do have connection with each other.
In this case we can conduct an inductive learning for CGNF .
Compared with semisupervised setting, the only difference of inductive learning is that we cannot see the test nodes in the training phase.
Thus we need to predict H tr only based on the training nodes X tr and their adjacency matrix A tr .
Then we can still use Eq.
13 for training.
For inference of the test labels, as we lose the connections between training nodes and test nodes, so the trainings labels cannot directly impact the inference of test labels.
In fact, we can still form a graph containing both training data and test data, however A i,j = 0 for training node i and test node j .
So the pairwise potential between training nodes and test nodes will also become 0 according to
Eq. 8.
So Eq.
14 is not useful anymore, because the pairwise potential becomes 0 if we just utilize the training labels for inference.
Instead, in this case we will use the DP-style inference.
E XPERIMENTS
Data We evaluated the effectiveness of CGNF on the following benchmark tasks: (1) classifying research topics using the Cora citation dataset (McCallum et al., 2000); (2) categorizing academic papers with the Pubmed database; (3) classify research areas using the Citeseer citation dataset (Sen et al., 2008); and (4) classifying protein functions across various biological protein-protein interaction (PPI) graph (Borgwardt et al., 2005).
Implementation details are in A.3 in Appendix.
[Table]
Baselines To evaluate the effectiveness of our method, we compare with the following baselines.
• GCN (Kipf & Welling, 2017) learns latent representations by encoding graph structure and node features.
It demonstrated signiﬁcant improvement in semi-supervised node classiﬁcation tasks.
• GraphSAGE (Hamilton et al., 2017) is an inductive variant of GCN which can generate node embedding for unseen data by sampling and aggregating features from a node’s local neighborhood.
• FastGCN
(Chen et al., 2018a) enhanced GCN with inductive ability and efﬁciency by interpreting graph convolutions as integral transforms of embedding function under probability measures.
Monte Carlo approaches were used to estimate integrals and lead to batched training scheme.
• DeepWalk (Perozzi et al., 2014) learns node embedding by extracting node sequence from graph using truncated random walks.
Skip-gram method is also utilized to maximum the conditional probability of neighbor nodes given context node.
It is a state-of-the-art graph embedding method that showed effectiveness on large-scale social networks classiﬁcation datasets.
• LINE
(Tang et al., 2015b) is the follow-up method of DeepWalk.
It considers both ﬁrst-order proximity and second-order proximity by maximizing the joint probability between nodes.
• Node2Vec
(Grover & Leskovec, 2016b) is a variant of DeepWalk by introducing two additional hyper-parameters q, p to control transition probability of random walk.
They approximate best ﬁrst search and depth ﬁrst search behavior for learning more informative representations.
For semi-supervised learning, we used the same setting for all the baselines, i
e. training the node embedding using all node features from both training data and test data.
While for inductive learning, we use node features only from training data.
4. R ESULTS
As shown in Table 3, CGNF is compared to a variety of high-performing baselines on a selection of benchmark graph node classiﬁcation tasks.
Results show CGNF models achieve the highest accuracy among all baselines with respect for all semi-supervised scenarios.
For skip-gram based methods including DeepWalk, Node2Vec and LINE, they perform worse than
GCN-based models due to these methods are all inherently unsupervised learning methods with post-processing supervised classiﬁers.
As a contrast, GCN-based methods combine semi- or supervised classiﬁcation which can embed richer graph information to yield better performance.
Notice that since GraphSAGE learns inductively, on Cora, Pubmed, and Citeseer it yields slightly lower accuracy than GCN.
Among these GCN-based methods, GraphSAGE is designed for inductive learning, thus performs best in the inductive task.
However, under semi-supervised setting (e.g., on Cora, Pubmed and
Citeseer datasets), it received very low accuracy due to having fewer labeled data.
Moreover, in inductive setting, skip-gram based methods require expensive additional training to inference on unseen test data which are inherently transductive.
CGNF outperforms GCN and FastGCN since label correlations are useful in bridging training data and unseen graphs. [Table]
R ELATED W ORKS
The connectivity-based node classiﬁcation approaches operate on the graph structure alone.
Among the most successful connectivity-based methods are node embedding techniques including (Perozzi et al., 2014; Grover & Leskovec, 2016a; Tang et al., 2015a; Hamilton et al., 2017; Kipf & Welling, 2016).
An underlying assumption of these techniques is that nodes which are closely connected in the graph, should have similar labels.
In the feature-based setting the graph structure can be incorporated in different ways, for instance by using regularization (Zhu et al., 2003; Belkin et al., 2006), combining attributes with node embeddings, or aggregating them over local neighborhoods (Defferrard et al., 2016; Kipf & Welling, 2017; Chen et al., 2018a).
Incorporating probabilistic graphical model like conditional random ﬁelds (CRF) and neural networks is one possible approach to tackle structured prediction task in different domains.
Initial works such as CNF (Peng et al., 2009) and NeuroCRF (Do & Artieres, 2010) yielded powerful models by adding neural networks layer after CRF’s input.
Similar idea shown in (Chen et al., 2018b) which applied CRF inference as a post-processing step after the training of deep convolutional neural networks for image segmentation task.
Another line of works tried to integrate CRF with deep neural networks in end-to-end form see (Zheng et al., 2015; Ma & Hovy, 2016; Long et al., 2018; Vemulapalli et al., 2016).
For example, (Zheng et al., 2015) formulated CRF with
Gaussian pairwise potentials and mean-ﬁeld approximate inference (Kr ¨ ahenb ¨ uhl & Koltun, 2011) in recurrent neural networks way in image segmentation task, and for sequence labeling tasks, (Ma &
Hovy, 2016) used a sequential CRF on top of DNNs to jointly decode labels for the whole sentence.
In addition, in Appendix, we also built a connection between CGNF and the graph based semisupervised models.
C ONCLUSION
In this paper we propose CGNF that extends GCN by deﬁning an energy function to capture the dependencies between labels for better node embedding and classiﬁcation.
We proposed efﬁcient ways to make the learning of CGNF tractable.
Compared with GCN, the proposed model gained signiﬁcant performance increase.
Future direction includes exploring other efﬁcient approximation method for training (such as mean-ﬁeld inference) and applying this energy based framework to other graph neural networks.
A A PPENDIX A. C ONNECTION TO G RAPH - BASED S EMI - SUPERVISED M ODEL
In some graph based semi-supervised learning models, the dependency between labels are also considered.
In this section, we will build some connection between CGNF and these graph-based semisupervised learning models.
We will show some graph regularization term can be transformed to one special form of our pairwise potential.
Take Zhu et al.
as an example.
They proposed the harmonic energy minimization method by minimizing the quadratic energy function Eq. 15.
[Equation]
Comparing to (13), we use p ( y i |
h i ) to replace the L 2 loss, and use U y i ,y j to take place of ( f i − f j ) 2 ; w i,j corresponds to the weight ˆ
A
i,j . U
y
i ,y j is an unknown parameter, so we have to consider the normalizer in the training phase if we use negative log likelihood as the objective function.
However, if we ﬁx U =
I , then the sum of the pairwise potential will become:
[Equation]
where ζ ( y i = y j )
=
if y i =
y j , and ζ ( y i = y j )
=
if y i (cid:54) =
y j ; || .
|| 22 is the square of the L 2 norm.
As ˆ A is a constant, optimizing this energy will be identical to the Eq. 15.
In addition, the graph based semi-supervised model only considers label consistency.
However, by deﬁning correlation matrix U , the label correlation integrated in our CGNF is much more ﬂexible.
A. I MPLEMENTATION D ETAILS
For Cora, Pubmed, and Citeseer, we follow the experiment setup in (Kipf & Welling, 2017) to demonstrate the effective use of CGNF .
For PPI, we consider the node classiﬁcation task in inductive semi-supervised setting.
Thus, PPI (inductive) is conducted where nodes in validation and test dataset are totally unseen.
All methods are trained in training data; hyperparameters (e.g. dropout rate for all layers, number of hidden units) are only optimized on Cora based on the performance on validation dataset and are used for the rest datasets.
We report the micro average F1 measure which is the same as accuracy on multiclass tasks.
We showed the speciﬁc dataset split ratio in Table.
2 which keeps the labels for training data in a small portion to test the methods in semi-supervised setting.
For DeepWalk, Node2Vec, the number of random walks to start at each node is set to 10 and window size of skip-gram model is set to 10 by default.
We choose the 2nd-order relation for LINE and save the embeddings of the best validation accuracy during training.
One-vs-Rest technique is used based on logistic regression implemented using scikit-learn (Pedregosa et al., 2011) for DeepWalk, LINE and Node2Vec.
We train GCN-based methods for 200 epochs using Adam (Kingma & Ba, 2014) with a learning rate of .01 and early stopping with a window size of 30, i.e., we stop training if the current performance in validation set is lower than average of 30 consecutive epochs.
They are implemented with two layers initialized using Glorot method (Glorot & Bengio, 2010) and appended with dropout (Srivastava et al., 2014) for each layer.
We report mean performance of 10 runs with random weight initializations for GCN-based methods.
In detail, we set .5 dropout rate, 100 hidden size, 5 ∗ 10 − 4 L2 regularization, 1 mixture weight γ for
CGNF and its variants.
We compare against other GCN-based methods where we choose their best performing model or hyper-parameters setting as claimed in their paper.