GLAM: G RAPH L EARNING BY M ODELING A FFINITY TO L ABELED N ODES FOR G RAPH N EURAL N ETWORKS
A BSTRACT
We propose GLAM, a semi-supervised graph learning method for cases when there are no graphs available.
This approach learns a graph as a convex combination of the unsupervised k -Nearest Neighbor graph and a supervised label-afﬁnity graph.
The latter graph directly captures all the nodes’ label-afﬁnity with the labeled nodes i.e. how likely a node has the same label as the labeled nodes.
Our experiments show that GLAM gives close to or better performance (up to ∼ 1.5%) while being simpler and faster (up to 70x) to train than state-of-the-art graph learn- ing methods.
We also demonstrate the importance of individual components and contrast them with the state-of-the-art methods.
I NTRODUCTION AND R ELATED W ORK
Incorporating graph to improve semi-supervised classiﬁcation is a long and well-studied problem.
Early methods used graph to propagate labels (Zhu & Ghahramani, 2002;
Zhou et al., 2004).
Classiﬁers later utilized graphs for regularization (Belkin et al., 2006; Weston et al., 2008).
Graph Neural Networks (GNNs), which incorporate graphs in their architecture, was introduced recently. Graph
Convolutional Network (GCN) is the most prominent among them (Kipf & Welling, 2017; Wu et al., 2019).
GNN models have shown performance exceeding approaches that only use the graph to propagate labels or to regularize.
However, all these approaches assume the existence of a graph.
In this paper, we are mainly interested in scenarios where we do not have access to a graph.
In the absence of a graph, utilizing the k -Nearest Neighbor ( k NN) graph has been proven to be quite effective (Belkin et al., 2006; Gidaris & Komodakis, 2019) and gives improvements over models that do not use graph information.
Recent literature, including LDS (Franceschi et al., 2019) and IDGL (Chen et al., 2020), propose graph learning approaches beyond simple k NN construction.
LDS treats a graph as a hyperparameter and employs a bi-level optimization setup. IDGL learns a graph by iteratively learning latent node representations and construct adjacency structures from these representations.
PGLearn (Wu et al., 2018), TO-GCN (Yang et al., 2019), and GCRN (Yu et al., 2020) explored similar ideas to LDS and IDGL.
Tangential to these works, the Graph Agreement Model (GAM) (Stretcu et al., 2019) proposed an agreement model to regularize the classiﬁer.
Bojchevski et al.
Trivedi et al.
(2020) explored generative models for graphs, but these approaches do not cater to graph learning.
Existing graph learning approaches are iterative and computationally expensive.
An alternative way to model edges is by learning attention over edges using models like GAT (Veliˇckovi´c et al., 2018) and SuperGAT (Kim & Oh, 2021).
However, learning attention across all the edges with limited labels is difﬁcult.
To address these shortcomings, we propose GLAM, a graph learning approach. GLAM combines a supervised graph with an unsupervised kNN graph.
We introduce a novel labelafﬁnity model, which uses an explicit loss function to learn the supervised graph.
Our experiments show that GLAM gives close to or better performance (up to ∼ 1.5%) while being simpler and faster (up to 70x) to train than state-of-the-art graph learning methods.
P ROBLEM S TATEMENT
In the semi-supervised classiﬁcation problem, we have labeled examples D
= { ( x i , y i ) } li =1 and unlabeled examples U = { x i } l + u i =
l +1 , where x i ∈ X .
Let Y be the set of all possible labels and y be the vector [ y 1 , y 2 , . . .
, y l + u ] .
The goal in traditional semi-supervised classiﬁcation problem is to learn a function f : X (cid:55)→ Y and it is often learnt by solving the following minimization problem,
[Equation]
where L is the loss function, and α is the regularization coefﬁcient.
In Graph Neural Networks (GNNs), the classiﬁer function additionally assumes access to a graph between the data points in X .
Let this graph be G . Then, this classiﬁer function is often estimated by solving,
[Equation]
But, often in practice, we may not have access to any graphs. The goal of our work here is to estimate a classiﬁer function by solving, [Equation] where Ω is additional loss terms which could depend on G and the known labels y and β is its coefﬁcient.
P ROPOSED A PPROACH
The proposed model consists of three basic components: Label-Afﬁnity Model, Convex Combination of Graphs and GCN.
GLAM’s architecture is presented in Figure
Label-Afﬁnity Model: Attention-based models attempt to learn attention over all the edges of a graph.
This could be challenging when working with limited labeled data.
The critical insight here is to realise that having a few noisy edges can hurt more than throwing away several good edges (i.e. edges where the source and target nodes have same labels).
To illustrate this, we remove all the noisy edges from the k NN graph and perform two experiments where we vary the percentage of noisy/good edges added/removed, and report GCN performance on these graphs.
We observe that adding noisy edges hurts performance more than removing several good edges.
A detailed analysis of this experiment is presented in the Appendix A.4.
Based our analysis, we make an assumption that more conﬁdent predictions can be made on edges from any node to labeled node, instead of any arbitrary edge, because of the partial information available.
Conventional attention-based models attempt to learn attention as a function X × X (cid:55)→ R . However, we decided to change the model to X (cid:55)→ ∆ | D | where X is the set of instances, ∆ is the probability simplex: { (cid:80) | D | i =1 θ i = 1 and ∀ i θ i ≥ 0 } . D is the set of labeled data as indicated in Section 2.
This form of modeling was a straight-forward way of enforcing our restriction.
This model predicts a distribution over the labeled set. The probability value indicates how likely it is for the target labeled node to have the same label as the input node.
We model this with a simple two-layer neural network as follows: [Equation]
where X ∈ R ( l + u ) × d is the feature matrix, d is the number of features, σ 1 is a non-linear activation function and W 1 , W 2 are model weights.
There are two advantages to this model - 1] it is a simple way of enforcing restriction on edges that we desire, 2] it also allows us to write an explicit loss function using the labeled data (we will discuss this in more detail below).
The label-afﬁnity model can be used to construct the afﬁnity graph. To construct a graph, for every node, we compute its predicted distribution over the labeled set and add the edge with the highest probability. This edge is added in both directions. We can write this mathematically as,
[Equation]
In Equation 5, we get the highest afﬁnity labeled node for all the nodes.
In Equation 6, we augment the remaining unlabeled columns which are simply zeroes to complete the full graph.
In Equation 7, we make the graph symmetric. However, since A RG -M AX is not differentiable, we use the GumbelSoftmax trick Jang et al. and generate samples from Gumbel-Softmax distribution instead to create the graph.
Relying on the classiﬁer loss to learn the parameters would be difﬁcult as the gradients that ﬂow into this model are minuscule.
Thereby, we add explicit loss for this model by constructing labeled data from D .
For every x i ∈ D , we create a new label vector y Ai such that,
[Equation]
where y Ai [ j ] is the j th element of the vector.
Further, it is normalized as (cid:99) y Ai = y
Ai / 1 (cid:62) y Ai .
We use this to deﬁne the loss for the afﬁnity model as follows: [Equation] where Z Ai is the row corresponding to the i th point in X .
Combining Graphs and GCN: The constructed afﬁnity graph, G A , contains only a few high conﬁdent edges and restricts feature propagation, particularly between unlabeled nodes.
Thereby, we utilize the k NN graph to assist in this regard.
The k NN graph is constructed from input features.
We use cosine distance metric in our experiments.
Number of neighbors, k , is a hyper-parameter. We pre-process the k NN graph by cropping all the incoming edges to the labeled nodes to reduce noise among labeled nodes.
We recognize that there is noise in other edges of the graph, but it is difﬁcult to ﬁlter this noise.
However, removing incoming noise to training nodes gives improvements as we show in Table 3 (a).
We refer to this graph as G ck , where ck stands for cropped k NN graph.
This graph is then combined with the afﬁnity graph as follows:
[Equation]
This combined graph is ﬁnally fed to a two-layer GCN model. However, because of the cropped k NN graph, this combined graph is asymmetric. For computing the Laplacian for GCN, we can use either the indegree or the outdegree matrix.
In our experiments, we have utilized the indegree matrix.
The combined graph is ﬁnally fed to a two-layer GCN model and the ﬁnal objective we minimize is,
[Equation]
where α A , α C are regularization coefﬁcients, W ia and W igcn are afﬁnity model and GCN model weights.
The ﬁrst term in Equation 11 refers to the classiﬁer loss, which takes the combination of unsupervised k NN and supervised afﬁnity graph (Equation 10).
The second term is the afﬁnity loss (Equation 9) which models label afﬁnity from all nodes to labeled nodes.
The afﬁnity loss together with classiﬁer loss improve afﬁnity predictions. Improvement in afﬁnity predictions improves classiﬁer performance. [Table] Note on Input Features: Note that, for the construction of the k NN graphs in previous section the input to the afﬁnity model in Equation 4, we have used X (the feature matrix).
However, in a recent work on text classiﬁcation, the authors of HeteGCN Ragesh et al.
(2021) have proposed an architecture which utilizes a normalized version of X (cid:62) X , call it (cid:93) X (cid:62) X as a ﬁrst layer of GCN.
Essentially they are utilizing feature correlation as the second layer graph.
The simpliﬁed version of this model is simply equivalent to X (cid:93) X (cid:62) X . We believe that the HeteGCN model beneﬁts from this feature correlation matrix and that utilizing X (cid:93) X (cid:62) X instead of X should beneﬁt our graph learning problem as well.
We simply refer to them as boosted features. We use these boosted features in k NN construction and as input to afﬁnity model.
We show experimental results with both the normal and the boosted features and not for just our model but for all the baselines, to illustrate how these simple modiﬁed features can give signiﬁcant performance improvements.
E XPERIMENTS
We conducted several experiments to highlight the effectiveness of GLAM against several baselines and state-of-the-art graph learning methods (refer to appendix A.2 for details).
We discovered a label-leakage bug in GAM’s source code, which post ﬁxing shows marginal improvements over baselines.
A comparison against bug-ﬁxed version of GAM is reported in Appendix A.3.
We restrict our setting to semi-supervised node classiﬁcation problems where a graph structure is not available.
Our experiments also present empirical analysis to illustrate why our model performs better.
Additional experiments are reported in Appendix A.3.
Performance Improvement and Speed up .
We report the mean test accuracy and standard deviation over ﬁve random seeds for benchmark datasets (refer to appendix A.1 for details).
GLAM shows competitive performance across all benchmark datasets, except PubMed, with accuracy gains of up to 3%.
We further improve all the baselines by feeding them with boosted features either as input features or by using them for constructing k NN graphs.
Our results in Table 1 show that these new features, in most cases, considerably improve baseline models with accuracy gains of up to 14%.
We perform a timing comparison by computing the average end-to-end training time over 2000 runs.
Table 3 (b) shows timing analysis for different benchmark datasets.
We observe that we get up to ∼ 42 × speedup compared to IDGL and up to ∼ 70 × speed up compared to LDS.
Analysis .
What do Attention models capture?
We can view GLAM as an instance of an attention model that places sparse attention on edges between all nodes and labeled nodes.
SuperGAT (Kim & Oh, 2021) works under the assumption that if two nodes are linked, they are more relevant and suggests that if the homophily of the graph is > 0 . 2 , SuperGAT MX performs well.
However, we observe that GAT performs as well as or better than SuperGAT MX with k NN graphs.
We report homophily percentages for k NN graphs in Table 4, indicating that there might be more underlying reasons for when attention works or does not.
Towards this end, and to quantify the quality of learned attention coefﬁcients, we compute few metrics.
(1) Bad Neighbor Ratio (BNR):
We deﬁne bad neighbors for a center node as all the neighboring nodes having different labels than the center node.
For Attention-based models, we extract attention matrices (A) from the hidden layers for all heads.
For GLAM and GCN
we treat Laplacian matrices as attention matrices
We compute BNR as follows: [Table]
[Table]
in (b).
[Equation]
where, BW ( i ) =
(cid:80) j ∈ N
[Equation]
i ) A
[Equation] ) ij [Equation] =
[Equation] j ) , GW [Equation] i ) =
[Equation]
j ∈ N ( i ) A ( h ) ij 1 (cid:96) ( i )=
(cid:96) ( j ) .
In the above equation, the number of attention heads is denoted by h, attention matrix by A, and the total number of nodes by | V | .
This metric gives a direct insight into how the underlying attention mechanism performs.
(2) Weighted homophily: this is a simple extension of homophily.
It is equal to the ratio of attention placed on all good edges to the attention placed on all edges.
Table 2 shows these metrics for models in comparison.
We ﬁnd that the Bad Neighbor Ratio correlates well with model performance giving us a perspective on why GLAM performs better.
(3) To study the importance of Label Afﬁnity Graph, we computed the average weight placed on edges in the afﬁnity graph and report it in Table 4.
We notice that, in most datasets, W A > 0.3 is assigned to afﬁnity edges, implying the importance of them.
In the PubMed dataset, GLAM places signiﬁcant weight on the k NN graph, which indicates that when k NN graph quality is good, they are preferred over afﬁnity graphs.
[Table]
Ablation Study .
Table 3 (a) shows that removing the afﬁnity graph and afﬁnity loss signiﬁcantly affects the performance, thus indicating their importance.
GLAM’s performance with only cropped k NN (w/o afﬁnity graph in Table 3 (a)) is equivalent to that of GCN with k NN (Table 1).
Therefore, removing all incoming edges to the training nodes does not affect and sometimes improves the performance.
Another observation is that unless there is an explicit loss for the afﬁnity term, we see minuscule improvements over plain k NN-GCN model (Table 1).
D ISCUSSION
In this paper, we proposed a model to jointly learn the graph and classiﬁer for semi-supervised classiﬁcation tasks where no graphs are available.
Our experimental results suggest that our model is fast to train and particularly effective when the unsupervised k NN graph is noisy.
We analysed and compared with baselines along several dimensions, highlighting their limitations and how our model addresses them.
For future work, we want to explore combining the merits of IDGL with our work and see if a single model can work across all possible graphs with different noise levels.
Also of great interest would be to combine the proposed idea with some of the graph generative models like NetGAN (Bojchevski et al., 2018) and GraphOpt (Trivedi et al., 2020).
A A PPENDIX A. D ATASETS
We evaluate on ﬁve citation network datasets: Cora, CiteSeer, PubMed taken from Jaafor & Birregah (2017), and ACM, DBLP taken from Wang et al.
In the DBLP dataset, each node represents an author.
In rest of the datasets, nodes correspond to documents (scientiﬁc papers).
The node features correspond to sparse bag-of-words features with either binary or TF-IDF values.
These datasets are evaluated for node classiﬁcation task in a transductive setting closely following the experimental setup of Yang et al.
For Cora, CiteSeer, and PubMed datasets, we use the standard split from previous work (Kipf & Welling, 2017).
For ACM and DBLP datasets, we ﬁx the validation and test set to 500 and 1000 nodes and create a training set by sampling 20 nodes per class from the remaining nodes to be in line with the standard split. Additionally, we also evaluate on three text classiﬁcation datasets where a graph is not available with the given dataset. 20NG consists of long text documents, categorized into 20 newsgroups. MR consists of movie reviews classiﬁed into positive and negative sentiments.
Ohsumed consists of medical abstracts, which are categorized into 23 cardiovascular diseases (Yao et al., 2019).
Detailed statistics of the datasets used are available in the Table 5. [Table]
[Table]
A. B ASELINES AND H YPER - PARAMETERS R ANGES
We compare GLAM against several baselines covering simple non-graph based approaches, semisupervised classiﬁcation methods, graph neural networks, and state-of-the-art graph learning approaches.
Below are the hyper-parameters ranges we followed for tuning the baselines.
For all the methods that rely on graphs, we construct k NN graphs from input features using cosine metric.
Number of neighbors, k , is a hyper-parameter and swept over { 5, 10, 15, 20 } .
LogReg: Logistic Regression’s weight-decay hyper-parameter, C, is tuned over [1e-4, 1e4] in powers of 10 on the validation set.
MLP: We employ a Multi Layer Perceptron with 1 hidden layer.
For tuning, hidden layer dimensions were swept over { 32, 64, 128 } , weight decay over [1e-4, 1e4] in logarithmic steps, learning rate over [1e-3, 1e-2, 1e-1], and dropout from (0, 1).
LP: Zhu & Ghahramani (2002)
In Label Propagation, we tune the hyper-parameter α (clamping factor) over the range (0, 1) in steps of 0.01.
We observe much better results for LP than reported in
LDS because of this extensive tuning.
ManiReg: Belkin et al.
Manifold regularization’s hyper-parameters γ A and γ I are tuned from the range (1e-5, 1e2) in logarithmic steps.
We observe a discrepancy in ManiReg’s numbers reported in LDS.
Kipf & Welling (2017) reported 59.5 and 60.1 as test performance on Cora and
Citeseer datasets using the original graphs that are part of the datasets.
LDS reports 62.3 and 67.7 as mean test accuracy for these datasets using k NN graphs.
However, k NN graphs are of poor quality in terms of homophily and GNN performance on these datasets.
SemiEmb: Weston et al.
Semi-Supervised embedding’s hyper-parameters λ is tuned over (1e-5, 1e2) in logarithmic steps, hidden layer dims over { 32, 64, 128 } , weight decay over [1e-4, 1e4] in steps of 10, learning rate from { 1e-3, 1e-2, 1e-1 } , and dropout from (0, 1).
GCN: Kipf & Welling (2017)
We used 2 layered Graph Convolutional Networks and follow the hyper-parameter ranges mentioned in Shchur et al
(2018) for tuning.
GAT : Veliˇckovi´c et al
For tuning Graph Attention Networks, we consulted Shchur et al.
(2018) for hyper parameters ranges.
SuperGAT : Kim & Oh we rely on authors 1 code for experiments and follow Kim & Oh (2021) for tuning.
LDS: Franceschi et al.
We rely on authors 2 code to perform LDS experiments on our benchmark datasets.
We follow the hyper-parameters ranges mentioned by the author.
IDGL : Chen et al
We rely on authors 3 code to perform IDGL experiments on our datasets.
For PubMed dataset, we ran IDGL-Anchor variant to report numbers and for the rest datasets, IDGL base variant was used for conducting experiments.
The hyper-parameters mentioned by the author are used for tuning the model.
CL-MLP : Cascante-Bonilla et al.
We implemented Curriculum Labeling algorithm, a very recent
SSL
method,
for an MLP as described by the authors. For tuning the MLP, we used the hyper parameters described above ICT-MLP : Verma et al.
Interpolation Consistency Training is one of the latest SSL method proposed.
We
implemented ICT algorithm for
an MLP followed
Verma et
al
for hyper parameter tuning
G3NN : Ma et al.
We rely on authors 4 code for experiments and sweep the hyper-parameters ranges suggested by the authors.
BGCN :
Zhang et al.
We rely on authors 5 code for our experiments and sweep the hyperparameters ranges suggested by the authors. GLAM: Hyper-parameters α A and α C are tuned from (1e-5, 1e4) in logarithmic steps, learning rate from (1e-3, 1e0), dropouts from (0, 1), W ck from (0, 1), afﬁnity classifer’s hidden layer dims from { 32, 64, 128, 256 } , GNN’s hidden layer dims from { 16, 32, 64, 128 } , k NN’s k from { 5, 10, 15, 20 } , and gumbel softmax’s temperature is set to 1e-10.
We use Adam optimizer to minimize our combined loss term.
GLAM is trained for 500 epochs with an early stopping criterion of no improvement in validation accuracy for 25 epochs.
Hyper-parameter tuning was done using optuna Akiba et al.
We swept through 2000 conﬁgurations using TPE sampler in optuna for each dataset.
All the models we implemented except LP and LogReg were written in Tensorﬂow Abadi et al.
LP and LogReg were implemented using scikit-learn python package Pedregosa et al.
For all models, test accuracy is reported for the conﬁguration that achieves the highest validation accuracy. A. A DDITIONAL R ESULTS
We compare GLAM against additional baselines including recent SSL methods, bug-ﬁxed version of GAM, Bayesian graph methods and additional results in Table A.3.
We also assess GLAM on datasets where a graph is genuinely not available along with the datasets.
Table A.3 shows that
GLAM consistently outperforms previous state-of-the-art methods on several of these datasets.
A. A DDITIONAL A NALYSIS
Noise Analysis .
We perform the following analysis to show that having a few noisy edges can hurt performance more than throwing away several good edges (i.e. edges where source and target nodes have same labels).
We construct k NN graphs for several benchmark datasets.
We throw away all the noisy edges from this graph and call this as Perfect-kNN graph. We conduct two experiments BGCN
[Table] [Figure] Plots - 1] We randomly add noisy edges to this Perfect-kNN graph, and 2] We randomly remove good edges from the Perfect-kNN graph.
We report test accuracies (using GCN) for varying percentages of added noisy edges and removed good edges.
Figure 2 shows these plots.
We observe that adding noisy edges deteriorates the performance of the model.
However, even after removing 50%-75% of good edges, the GCN can still perform well on the test set.
This observation suggests that it is important to reduce the amount of noise added rather than saving good edges.
GLAM v/s IDGL: Latent Node Representations .
IDGL is an iterative approach, thereby, lowquality graph leads to learning poor representations, and this effect cascades over multiple iterations leading to minor or no accuracy gains.
We note that IDGL performs better on PubMed compared to GLAM as the k nn graph has the homophily score close to the original link graph.
GLAM works better in the presence of noisy graphs.
Figures 3(b), 3(c) shows IDGL’s ﬁrst and last iteration’s embeddings and Figure 3(a) shows GLAM’s embeddings TSNE plots on the DBLP dataset.
We observe that the ﬁnal iteration’s plot is similar to the ﬁrst iteration plot whereas GLAM’s plot shows discernible clusters. [Figure] [Table] [Figure]
Is Afﬁnity Graph alone enough? .
Table 8 shows GCN’s performance on using only afﬁnity graph.
We see that using afﬁnity graph alone is not sufﬁcient.
Afﬁnity graph only contains edges from all nodes to labeled nodes restricting feature propagation, thus limiting the performance of GNNs.
We study how placing different weights on the afﬁnity graph during graph combination affects GCN’s
Kipf & Welling (2017) performance.
We observe that in all the datasets, adding afﬁnity graphs improves the performance of GNNs.
Figure 4 illustrates the effect of the afﬁnity graph on four different benchmark datasets.
A bell shape sort of behavior is observed in most cases, where the performance of GNN starts to dwindle as more weight is given to the afﬁnity graph.
K NN graphs are responsible for feature propagation.
Placing more weight on the afﬁnity graph affects feature propagation and we start losing performance after a certain point.