W HY P ROPAGATE A LONE ? P ARALLEL U SE OF L ABELS AND F EATURES ON G RAPHS
A BSTRACT
Graph neural networks (GNNs) and label propagation represent two interrelated modeling strategies designed to exploit graph structure in tasks such as node property prediction.
The former is typically based on stacked message-passing layers that share neighborhood information to transform node features into predictive embeddings.
In contrast, the latter involves spreading label information to unlabeled nodes via a parameter-free diffusion process, but operates independently of the node features.
Given then that the material difference is merely whether features or labels are smoothed across the graph, it is natural to consider combinations of the two for improving performance.
In this regard, we have recently proposed to use a randomly-selected portion of the training labels as GNN inputs, concatenated with the original node features for making predictions on the remaining labels.
This so-called label trick accommodates the parallel use of features and labels, and is foundational to many of the top-ranking submissions on the Open Graph
Benchmark (OGB) leaderboard.
And yet despite its wide-spread adoption, thus far there has been little attempt to carefully unpack exactly what statistical properties the label trick introduces into the training pipeline, intended or otherwise.
To this end, we prove that under certain simplifying assumptions, the stochastic label trick can be reduced to an interpretable, deterministic training objective composed of two factors.
The ﬁrst is a data-ﬁtting term that naturally resolves potential label leakage issues, while the second serves as a regularization factor conditioned on graph structure that adapts to graph size and connectivity.
Later, we leverage this perspective to motivate a broader range of label trick use cases, and provide experiments to verify the efﬁcacy of these extensions.
I NTRODUCTION
Node property prediction is a ubiquitous task involving graph data with node features and/or labels, with a wide range of instantiations across real-world scenarios such as node classiﬁcation (Velickovic et al., 2018) and link prediction (Zhang & Chen, 2018), while also empowering graph classiﬁcation (Gilmer et al., 2017), etc.
Different from conventional machine learning problems where there is typically no explicit non-iid structure among samples, nodes are connected by pre-speciﬁed edges, and a natural assumption is that labels and features vary smoothly over the graph.
This smoothing assumption has inspired two interrelated lines of research: First, graph neural networks (GNNs) (Kipf & Welling, 2017; Hamilton et al., 2017; Li et al., 2018; Xu et al., 2018; Liao et al., 2019; Xu et al., 2019) leverage a parameterized message passing strategy to convert the original node features into predictive embeddings that reﬂect the features of neighboring nodes.
However, this approach does not directly utilize existing label information beyond their inﬂuence on model parameters through training.
Second, label propagation algorithms (LPA) (Zhu, 2005; Zhou et al., 2003; Zhang & Lee, 2006; Wang & Zhang, 2006; Karasuyama & Mamitsuka, 2013; Gong et al., 2017; Liu et al., 2019) spread label information via graph diffusion to make predictions, but cannot exploit node features.
As GNNs follow a similar propagation mechanism as the label propagation algorithm, the principal difference being whether features or labels are smoothed across the graph, it is natural to consider combinations of the two for improving performance.
Examples motivated by this intuition, at least to varying degrees, include APPNP (Klicpera et al., 2019), Correct and Smooth (C&S) (Huang et al.,
, GCN-LPA (Wang & Leskovec, 2020), and TPN (Liu et al., 2019).
While often effective, these methods are not all end-to-end trainable and easily paired with arbitrary GNN architectures.
And in a related fashion Jia & Benson (2021) introduce an elegant generative framework that uniﬁes LPA and
GNNs, although the parallel propagation of both features and labels is not explicitly considered.
Among these various combination strategies, our previously proposed label trick (Wang et al., 2021) has enjoyed widespread success in facilitating the parallel use of node features and labels via a stochastic label splitting technique.
In brief, the basic idea is to use a randomly-selected portion of the training labels as GNN inputs, concatenated with the original node features for making predictions on the remaining labels during each mini-batch.
The ubiquity of this simple label trick is evidenced by its adoption across numerous GNN architectures and graph benchmarks (Sun & Wu, 2020; Kong et al., 2020; Shi et al., 2021; Li et al., 2021).
And with respect to practical performance, this technique is foundational to many of the top-ranking submissions on the Open Graph Benchmark (OGB) leaderboard (Hu et al., 2020).
For example, at the time of this submission, the top 10 results spanning multiple research teams all rely on the label trick, as do the top 3 results from the recent KDDCUP 2021 Large-Scale Challenge
MAG240M-LSC (Hu et al., 2021).
And yet despite its far-reaching adoption, thus far the label trick has been motivated primarily as a training heuristic without a strong theoretical foundation.
Moreover, many aspects of its underlying operational behavior have not been explored, with non-trivial open questions remaining.
For example, while originally motivated from a stochastic perspective, is the label trick reducible to a more transparent deterministic form that is amenable to interpretation and analysis?
Similarly, are there any indirect regularization effects with desirable (or possibly undesirable) downstream consequences?
And how do the implicit predictions applied by the model to test nodes during the stochastic training process compare with the actual deterministic predictions
used
during inference?
If there is a discrepancy, then the generalization ability of the model could be compromised.
And ﬁnally, are there natural use cases for the label trick that have so far ﬂown under the radar and been missed?
We take a step towards answering these questions via the following two primary contributions:
• We prove that in certain restricted settings, the original stochastic label trick can be reduced to an interpretable, deterministic training objective composed of two terms: (1) a data-ﬁtting term that naturally resolves potential label leakage issues and maintains consistent predictions during training and inference, and (2) a regularization factor conditioned on graph structure that adapts to graph size and connectivity.
Furthermore, complementary experiments applying the label trick to a broader class of graph neural network models corroborate that similar effects exists in more practical real-world settings, consistent with our theoretical ﬁndings.
• Although in prior work the label trick has already been integrated within a wide variety of GNN models, we introduce novel use-cases motivated by our analysis.
This includes exploiting the label trick to: (i) train simple end-to-end variants of label propagation and C&S, and (ii) replace stochastic use cases of the label trick with more stable, deterministic analogues that are applicable to GNN models with linear propagation layers such as SGC (Wu et al., 2019), TWIRLS (Yang et al., 2021) and SIGN (Frasca et al., 2020).
Empirical results on node classiﬁcation benchmarks verify the efﬁcacy of these simple enhancements.
Collectively, these efforts establish a more sturdy foundation for the label trick, and in doing so, help to ensure that it is not underutilized.
B ACKGROUND
Consider an undirected graph G =
( V, E ) with n = | V | nodes, the node feature matrix is denoted by
X ∈ R n × d and the label matrix of the nodes is denoted by Y ∈ R n × c , with d and c being the number of channels of features and labels, respectively.
Let A be the (unweighted) adjacency matrix, D the degree matrix and S = D − 12 AD − 12 the symmetric normalized adjacency matrix.
The symmetric normalized Laplacian L can then be formulated as L
= I n − S .
We also deﬁne a training mask matrix as M tr = (cid:18) I m   (cid:19) n × n , where w.l.o.g.
we are assuming that the ﬁrst m nodes, denoted
D tr , form the training dataset.
We use P to denote a propagation matrix , where the speciﬁc P will be described in each context.
2. L ABEL P ROPAGATION A LGORITHM
Label propagation is a semi-supervised algorithm that predicts unlabeled nodes by propagating the observed labels across the edges of the graph, with the underlying smoothness assumption that two nodes connected by an edge are likely to share the same label.
Following Zhou et al.
; Yang et al.
(2021), the implicit energy function of label propagation is given by
[Equation]
where F is the smoothed labels, Y tr = M tr Y is the label matrix of training nodes, and λ ∈ (0 , 1) is a regularization coefﬁcient that determines the trade-off between the two terms.
The ﬁrst term is a ﬁtting constraint , with the intuition that the predictions of a good classiﬁer should remain close to the initial label assignments, while the second term introduces a smoothness constraint , which favors similar predictions between neighboring nodes in the graph.
It is not hard to derive that the closed-formed optimal solution of this energy function is given by
F ∗ =
P Y , where P =
(1 − λ )
( I n − λ S ) − 1 .
However, since the stated inverse is impractical to compute for large graphs, P Y is often approximated in practice via P ≈ (1 − λ ) (cid:80) ki =
λ i S i Y .
From this expression, it follows that F can be estimated by the more efﬁcient iterations F ( k +1)
=
λ SF ( k )
+ (1 − λ ) F (0) , where F (0) =
Y tr and for each k , S smooths the training labels across the edges of the graph.
2. G RAPH N EURAL N ETWORKS FOR P ROPAGATING N ODE F EATURES
In contrast to the propagation of labels across the graph, GNN models transform and propagate node features using a series of feed-forward neural network layers.
Popular examples include GCN (Kipf & Welling, 2017), GraphSAGE (Hamilton et al., 2017), GAT (Velickovic et al., 2018), and
GIN (Xu et al., 2019).
For instance, the layer-wise propagation rule of GCN can be formulated as
X ( k +1)
= σ ( SX ( k ) W ( k ) ) where σ ( · ) is an activation function such as ReLU, X ( k ) is the k -th layer node representations with X (0) =
X , and W ( k ) is a trainable weight matrix of the k -th layer.
Compared with label propagation, GNNs can sometimes exhibit a more powerful generalization capability via the interaction between discriminative node features and trainable weights.
C OMBINING L ABEL AND F EATURE P ROPAGATION
While performing satisfactorily in many circumstances, GNNs only indirectly incorporate groundtruth training labels via their inﬂuence on the learned model weights.
But these labels are not actually used during inference, which can potentially degrade performance relative to label propagation, especially when the node features are noisy or unreliable.
Therefore, it is natural to consider the combination of label and feature propagation to synergistically exploit the beneﬁts of both as has been proposed in Klicpera et al.
Liu et al.
; Wang & Leskovec (2020); Wang et al.
(2021); Shi et al.
; Huang et al.
One of the most successful among these hybrid methods is our previously proposed label trick (Wang et al., 2021), which can be conveniently retroﬁtted within most standard GNN architectures while facilitating the parallel propagation of labels and features in an end-to-end trainable fashion.
As mentioned previously, a number of top-performing GNN pipelines have already adopted this trick, which serves to establish its widespread relevance (Sun & Wu, 2020; Kong et al., 2020;
Li et al., 2021;
Shi et al., 2021) and motivates our investigation of its properties herein.
To this end, we formally deﬁne the label trick as follows:
Deﬁnition 1 (label trick)
The label trick is based on creating random partitions of the training data as in D tr = D in ∪D out and D in ∩D out
= ∅ , where node labels from D in are concatenated with the original features X and provided as GNN inputs (for nodes not in D in zero-padding is used), while the labels from D out serve in the traditional role as supervision.
The resulting training objective then becomes
[Equation]
where Y in ∈ R n × c , is deﬁned row-wise as y in,i = (cid:26)
y i if i ∈ D in otherwise for all i , the function f
( X , Y in ; W ) represents a message-passing neural network with parameters W and the concatenation of X and Y in as inputs, and (cid:96) ( · , · ) denotes a point-wise loss function over one sample/node.
At inference time, we then use the deterministic predictor f
[ X , Y tr ; W ] i for all test nodes i / ∈ D tr .
R ELIABLE R ANDOMNESS THOUGH THE L ABEL T RICK
Despite its widespread adoption, the label trick has thus far been motivated as merely a training heuristic without formal justiﬁcation.
To address this issue, we will now attempt to quantify the induced regularization effect that naturally emerges when using the label trick.
However, since the formal analysis of deep networks is challenging, we initially adopt the simplifying assumption that the function f from (2) is linear, analogous to the popular SGC model from Wu et al.
For simplicity of exposition, in Sections 3.1 and 3.2 we will consider the case where no node features are present to isolate label-trick-speciﬁc phenomena.
Later in Sections 3.3 and 3.4 we will reintroduce node features to present our general results, as well as considering nonlinear extensions.
3. L ABEL T RICK WITHOUT N ODE F EATURES
Assuming no node features X , we begin by considering the deterministic node label loss
[Equation]
where [ · ] i indicates the i -th row of a matrix P Y tr W is a linear predictor akin to SGC, but with only the zero-padded training label matrix Y tr as an input.
Additionally, P here can in principle be any reasonable propagation matrix, not necessarily the one associated with the original label propagation algorithm.
However, directly employing (3) for training suffers from potential label leakage issues given that a simple identity mapping sufﬁces to achieve the minimal loss at the expense of accurate generalization to test nodes.
Furthermore, there exists an inherent asymmetry between the predictions computed for training nodes, where the corresponding labels are also used as inputs to the model, and the predictions for testing nodes where no labels are available.
At a conceptual level, these issues can be resolved by the label trick, in which case we introduce random splits of D tr and modify (3) to
[Equation]
For each random split, the resulting predictor only includes the label information of D in (through Y in ), and thus there is no unresolved label leakage issue when predicting the labels of D out .
In practice, we typically sample the splits by assigning a given node to D in with some probability α ∈ (0 , 1) ; otherwise the node is set to D out .
It then follows that E
[ |D in | ] = α |D tr | and E [ Y in ]
=
α Y tr .
3. S ELF - EXCLUDED S IMPLIFICATION OF THE L ABEL T RICK
Because there exists an exponential number of different possible random splits, for analysis purposes (with later practical beneﬁts as well) we ﬁrst consider a simpliﬁed one-versus-all case whereby we enforce that |D out | = 1 across all random splits, with each node landing with equal probability in
D out .
In this situation, the objective function from (4) can be re-expressed more transparently without any expectation as
[Equation]
) where Y i represents a matrix that shares the i -th row of Y and pads the rest with zeros, and
C = diag( P ) .
This then motivates the revised predictor given by
[Equation]
Remark 1 From this expression we observe the intuitive role that C plays in blocking the direct pathway between each training node input label to the output predicted label for that same node.
In this way the predictor propagates the labels of each training node excluding itself, and for both training and testing nodes alike, the predicted label of a node is only a function of other node labels.
This resolves the asymmetry mentioned previously with respect to the predictions from (3).
Remark
It is generally desirable that a candidate model produces the same predictions on test nodes during training and inference to better ensure proper generalization.
Fortunately, this is in fact the case when applying (6), which on test nodes makes the same predictions as label propagation.
To see this, note that M te ( P − C )
Y tr = M te P Y tr , where M te =
I n − M tr is the diagonal mask matrix of test nodes and P Y tr is the original label propagation predictor.
Although ultimately (6) will serve as a useful analysis tool below, it is also possible to adopt this predictor in certain practical settings.
In this regard, C can be easily computed with the same computational complexity as is needed to approximate P as discussed in Section 2.1 (and for alternative propagation operators that are available explicitly, e.g., the normalized adjacency matrix,
C is directly available).
3. F ULL E XECUTION OF THE L ABEL T RICK AS A R EGULARIZER
We are now positioned to extend the self-excluded simpliﬁcation of the label trick to full execution with arbitrary random sampling, as well as later, the reintroduction of node features.
For this purpose, we ﬁrst deﬁne Y out =
Y tr − Y in , and also rescale by a factor of 1 /α to produce (cid:101) Y in
= Y in /α .
The latter allows us to maintain a consistent mean and variance of the predictor across different sampling probabilities.
Assuming a mean square error (MSE) loss as computed for each node via (cid:96) ( y , (cid:98) y )
=
||
y − (cid:98)
y
|| 22 (later we consider categorical cross-entropy), our overall objective is to minimize
[Equation]
(7) where M out is a diagonal mask matrix deﬁned such that Y out =
M out Y =
Y tr − Y in and the random splits follow a node-wise Bernoulli distribution with parameter α as discussed previously.
We then have the following:
Theorem 1 Deﬁne Γ = (cid:0) diag( P T P ) − C T C (cid:1) 12 Y tr .
Then the label trick loss from (7) satisﬁes
[Equation]
Note that (diag( P T P ) − C T C ) is a positive semi-deﬁnite diagonal matrix, and hence its real square root will always exist.
Furthermore, we can extend this analysis to include node features by incorporating the SGC-like linear predictor P XW x such that Theorem 1 can then naturally be generalized as follows:
Corollary 1 Under the same conditions as Theorem 1, if we add the node feature term P XW x to the label-based predictor from (7), we have that
[Equation]
The details of the proofs of Theorem 1 and Corollary 1 are provided in Appendix B.1.
This then effectively leads to the more general, feature and label aware predictor
[Equation]
where W = { W x , W y } .
These theoretical results reveal a number of interesting properties regarding how the label trick behaves, which we summarize as follows:
Remark
Although the original loss involves an expectation over random data splits that is somewhat difﬁcult to interpret, based on (9), the label trick can be interpreted as inducing a deterministic objective composed of two terms: (i)
The error accrued when combining the original node features with the self-excluded label propagation predictor from (6) to mitigate label leakage, and (ii)
An additional graph-dependent regularization factor on the model weights associated with the labels that depends on α (more on this below).
We can also easily verify from (10) that the model applies the same prediction to test nodes during both training and inference, consistent with Remark 2.
Remark
If the graph has no edges, then there is no chance for overﬁtting to labels and (cid:0) diag( P T P ) − C T C (cid:1) 12 = 0 shuts off the superﬂuous regularization.
In contrast, for a fully connected graph, the value of Γ can be signiﬁcantly larger, which can potentially provide a beneﬁcial regularization effect.
Additionally, given that Γ also grows larger with graph size (assuming edges grow as well), (cid:107) Γ W y (cid:107) 2 F scales proportionately with the data ﬁtting term, which is generally expected to increase linearly with the number of nodes.
Hence (9) is naturally balanced to problem size.
Remark
The splitting probability α in (9) controls the regularization strength.
Speciﬁcally, when α tends to zero, fewer labels are used as input to predict a large number of output labels, which may be less reliable, and corresponds with adding a larger regularization effect.
Additionally, it means placing more emphasis on the original node features and downplaying the importance of the labels as input in (9), which explains the addition of a penalty on W y .
Conversely, when α tends to one, more labels are used as input to predict the output and the model approaches the deterministic self-excluded label trick.
Speciﬁcally, for random splits where |D out | = 1 , the loss mimics one random term from the self-excluded label trick summation, while for the splits when |D out | = 0 , the contribution to the expectation is zero and therefore does not inﬂuence the loss.
Splits with |D out | > 1 will have very low probability, so this situation naturally corresponds with canceling out the regularization term.
Later in Section 3.4 we will extend these observations to general nonlinear models.
We now turn to the categorical cross-entropy loss, which is more commonly applied to node classiﬁcation problems.
While we can no longer compute closed-form simpliﬁcations as we could with
MSE, it is nonetheless possible to show that the resulting objective when using the original label trick is an upper bound on the analogous objective from the self-excluded label trick.
More speciﬁcally, we have the following (see Appendix B.3 for the proof):
Theorem 2 Under the same conditions as in Theorem 1 and Corollary 1, if we replace the MSE loss with categorical cross-entropy we obtain the bound
[Equation]
where CE S ( · , · ) denotes the sum of row-wise cross-entropy of S .
3. N ONLINEAR E XTENSIONS
When we move towards more complex GNN models with arbitrary nonlinear interactions, it is no longer feasible to establish explicit, deterministic functional equivalents of the label trick for general α .
However, we can still at least elucidate the situation at the two extremes where α → 0 or α →alluded to in Remark 5.
Regarding the former, clearly with probability approaching one, Y in will always equal zero and hence the model will default to a regular GNN, effectively involving no label information as an input.
In contrast, for the latter we provide the following:
Theorem 3 Let f GNN ( X , Y ; W ) denote an arbitrary GNN model with concatenated inputs X and Y , and (cid:96) ( y , ˆ y ) a training loss such that (cid:80) i ∈D out (cid:96) (cid:0)
y i , f GNN [ X , Y in ; W ] i (cid:1) is bounded for all D out .
It then follows that
[Equation]
The proof is given in Appendix B.4.
This result can be viewed as a natural generalization of (5), with one minor caveat: we can no longer guarantee that the predictor implicitly applied to test nodes during training will exactly match the explicit function f GNN [ X , Y tr ; W ] applied at inference time.
Indeed, each f GNN [ X , Y tr − Y i ; W ] i will generally produce slightly different predictions for all test nodes depending on i unless f GNN is linear.
But in practice this is unlikely to be consequential.
B ROADER U SE C ASES OF THE L ABEL T RICK
Although the label trick has already been integrated within a wide variety of GNN pipelines, in this section we introduce three novel use-cases motivated by our analysis.
4. T RAINABLE L ABEL P ROPAGATION
In Sections 3.1 and 3.2 we excluded the use of node features to simplify the exposition of the label trick; however, analytical points aside, the presented methodology can also be useful in and of itself for facilitating a simple, trainable label propagation baseline when we choose P as in Section 2.1.
The original label propagation algorithm from Zhou et al.
(2003) is motivated as a parameter-free, deterministic mapping from a training label set to predictions across the entire graph.
However, clearly the randomized label trick from Section 3.1, or its deterministic simpliﬁcation from Section
3.2 can be adopted to learn a label propagation weight matrix W .
The latter represents a reasonable enhancement that can potentially help to compensate for interrelated class labels that may arise, especially in multi-label settings.
In contrast, the original label propagation algorithm implicitly assumes that different classes are independent.
Beyond this, other entry points for adding trainable weights are also feasible such as node-dependent or edge-dependent weights (Wang & Leskovec, 2020), nonlinear weighted mappings (Kipf & Welling, 2017), step-wise weights for heterophily graphs (Yamaguchi et al., 2016), or weights for different node types for heterogeneous graphs (Schlichtkrull et al., 2018).
4. D ETERMINISTIC A PPLICATION TO GNN S WITH L INEAR P ROPAGATION L AYERS
Many prevailing GNN models follow the architecture of message passing neural networks (MPNNs).
Among these are efﬁcient variants that share node embeddings only through linear propagation layers.
Representative examples include SGC (Wu et al., 2019), SIGN (Frasca et al., 2020) and TWIRLS (Yang et al., 2021).
We now show how to apply the deterministic label trick algorithm as introduced in Section 3.2 with the aforementioned GNN methods.
We begin with a linear SGC model.
In this case, we can compute ( P − C ) Y tr beforehand as the self-excluded label information and then train the resulting features individually without graph information, while avoiding label leakage problems.
And if desired, we can also concatenate with the original node features.
In this way, we have an algorithm that minimizes an energy function involving both labels and input features.
Additionally, for more complex situations where the propagation layers are not at the beginning of the model, the predictor can be more complicated such as f ( X , Y ; W ) =
[Equation]
13) where [ · , · ] denotes the concatenation operation, P =
[ P 0 , P 1 , . .
, P k − 1 ] T is the integrated propagation matrix, C
=
[diag( P 0 ) , diag( P 1 ) , . . .
, diag( P k − 1 )]
T , h 0 and h 1 can be arbitrary nodeindependent functions, typically multi-layer perceptrons (MLPs).
4. T RAINABLE C ORRECT AND S MOOTH
Correct and Smooth (C&S) (Huang et al., 2021) is a simple yet powerful method which consists of multiple stages.
A prediction matrix ˜ Y is ﬁrst obtained whose rows correspond with a prediction from a shallow node-wise model.
˜ Y is subsequently modiﬁed via two post-processing steps, correct and smooth , using two propagation matrices { P c , P s } , where typically P i = (1 − λ i )
( I n − λ i S ) − 1 , i ∈ { c, s } .
For the former, we compute the difference between the ground truth and predictions on the training set as E =
Y tr − ˜ Y tr and then form ˜ E
= γ ( P c E ) as the correction matrix , where γ ( · ) is some row-independent scaling function.
The ﬁnal smoothed prediction is formed as f C & S ( ˜ Y )
=
P s ( Y tr + M te ( ˜ Y + ˜ E )) .
This formulation is not directly amenable to end-to-end training because of label leakage issues introduced through Y tr .
In contrast, with the label trick, we can equip C&S with trainable weights to further boost performance.
To this end, we ﬁrst split the training dataset into D in and D out as before.
Then we multiply ˜
E in = γ
( P c ( Y in − ˜ Y in )) , the correction matrix with respect to Y in , with a weight matrix W c .
We also multiply the result after smoothing with another weight matrix W s .
Thus the predictor under this split is
[Equation]
where W = { ˆ W c , W s } and ˆ W c
= W c W s .
The objective function for optimizing W is
[Equation]
Since this objective resolves the label leakage issue, it allows end-to-end training of C&S with gradients passing through both the neural network layers for computing ˜ Y and the C&S steps.
At times, however, this approach may have disadvantages, including potential overﬁtting problems or inefﬁciencies due to computationally expensive backpropagation.
Consequently, an alternative option is to preserve the two-stage training.
In this situation, the base prediction in the ﬁrst stage is the same as the original algorithm; however, we can nonetheless still train the C&S module as a post-processing step, with parameters as in (14).
[Table] [Table]
E XPERIMENTS
As mentioned previously, the effectiveness of the label trick in improving GNN performance has already been demonstrated in prior work, and therefore, our goal here is not to repeat these efforts.
Instead, in this section we will focus on conducting experiments that complement our analysis from
Section 3 and showcase the broader application scenarios from Section 4.
The label trick can actually be implemented in two ways.
The ﬁrst is the one that randomly splits the training nodes, and the second is the simpler version introduced herein with the deterministic one-versus-all splitting strategy, which does not require any random sampling.
To differentiate the two versions, we denote the stochastic label trick with random splits by label trick (S) , and the deterministic one by label trick (D) .
The latter is an efﬁcient way to approximate the former with a higher splitting probability α , which is sometimes advantageous in cases where the training process is slowed down by high α .
Accordingly, we conduct experiments with both, thus supporting our analysis with comparisons involving the two versions.
We use four relatively large datasets for evaluation, namely Cora-full, Pubmed (Sen et al., 2008), ogbn-arxiv and ogbn-products (Hu et al., 2020).
For Cora-full and Pubmed, we randomly split the nodes into training, validation, and test datasets with the ratio of 6:2:2 using different random seeds.
For ogbn-arxiv and ogbn-products, we adopt the standard split from OGB (Hu et al., 2020).
We report the average classiﬁcation accuracy and standard deviation after 10 runs with different random seeds, and these are the results on the test dataset when not otherwise speciﬁed.
See Appendix C for further implementation details.
5. T RAINABLE L ABEL P ROPAGATION
We ﬁrst investigate the performance of applying the label trick to label propagation as introduced in (2) in the absence of features, and compare it with the original label propagation algorithm.
Tableshows that the trainable weights applied to the label propagation algorithm can boost the performance consistently.
Given the notable simplicity of label propagation, this represents a useful enhancement.
5. D ETERMINISTIC L ABEL T RICK APPLIED TO GNN S WITH L INEAR L AYERS
We also test the deterministic label trick by applying it to different GNN architectures involving linear propagation layers along with (13).
Due to the considerable computational effort required to produce P and C with large propagation steps for larger graphs, we only conduct tests on the
Cora-full, Pubmed and ogbn-arxiv datasets, where the results are presented in Table 3.
[Table]
From these results we observe that on Pubmed and ogbn-arxiv, the deterministic label trick boosts the performance consistently on different models, while on Cora-full, it performs comparably.
This is reasonable given that the training accuracy on Cora-full (not shown) is close to 100%, in which case the model does not beneﬁt signiﬁcantly from the ground-truth training labels, as the label information is already adequately embedded in the model.
5. E FFECT OF S PLITTING P ROBABILITY
In terms of the effect of the splitting probability α , we compare the accuracy results of a linear model and a three-layer GCN on ogbn-arxiv as shown in Figure 1.
For the linear model, α serves as a
[Figure]
regularization coefﬁcient.
More speciﬁcally, when α tends to zero, the model converges to the one without the label trick, while when α tends to one, it converges to the case with the self-excluded label trick.
For the the nonlinear GCN, α has a similar effect as predicted by theory for the linear models.
As α decreases, the model converges to that without the label trick.
Moreover, a larger α is preferable for linear models that may not require strong regularization like more complex GNNs.
5. T RAINABLE C ORRECT AND S MOOTH
We also verify the effectiveness of our approach when applied to Correct and Smooth (C&S), as described in Section 4.3.
Due to the signiﬁcant impact of end-to-end training on the model, it is not suitable for direct comparison with vanilla C&S for a natural ablation.
Therefore, in this experiment, we train the C&S as post-processing steps.
In Table 4, we report the test and validation accuracy, showing that our method outperforms the vanilla C&S on Cora-full and Pubmed.
And for ogbn-arxiv and ogbn-products, the trainable C&S performs better in terms of validation accuracy while is comparable to vanilla C&S in terms of test accuracy.
In principle, C&S can be applied to any base predictor model.
To accommodate tabular node features, we choose to use gradient boosted decision trees (GBDT), which can be trained end-to-end using methods such as Chen et al.
; Ivanov & Prokhorenkova (2021) combined with the label trick to avoid data leakage issues as we have discussed.
For evaluation, we adopt the four tabular node regression data sets from Ivanov & Prokhorenkova (2021) and train using the approach from
Chen et al.
Results are shown in Table 2, where the label trick can signiﬁcantly improve performance.
[Table]
C ONCLUSION
In this work we closely examine from a theoretical prospective our recently proposed label trick
[Table]
which enables the parallel propagation of labels and features and beneﬁts various SOTA GNN architectures, and yet has thus far not be subject to rigorous analysis.
In ﬁlling up this gap, we ﬁrst introduce a deterministic self-excluded simpliﬁcation of the label trick, and then prove that the full stochastic version can be regarded as introducing a regularization effect on the self-excluded label weights.
Beyond this, we also discuss broader applications of the label trick with respect to: (i)
Facilitating the introduction of trainable weights within graph-based methods that were previously either parameter-free (e.g., label propagation) or not end-to-end (e.g., C&S), and (ii) Eliminating the effects of randomness by incorporating self-excluded propagation within GNNs composed of linear propagation layers.
We verify these applications and evaluate the performance gains against existing approaches over multiple benchmark datasets.
Acknowledgments.
The Shanghai Jiao Tong University Team is supported by Shanghai Municipal
Science and Technology Major Project (2021SHZDZX0102) and National Natural Science Foundation of China (62076161, 62177033).
We would also like to thank Wu Wen Jun Honorary Doctoral
Scholarship from AI Institute, Shanghai Jiao Tong University
A A DDITIONAL E XPERIMENTS A. E FFECT OF T RAINING A CCURACY
Intuitively
[Table]
the label trick introduces more information about the ground-truth labels in the training set to the model.
In this experiment, we investigate the role of training accuracy in training with the label trick.
We produce two new datasets by projecting the input features on Cora-full and Pubmed to 32 dimensions (i
[Table]
e., Cora-full-32d and Pubmed-32d, respectively) via PCA.
This is a much more challenging task because less feature information is available for prediction.
We train SGC, SIGN and TWIRLS under the same settings as in Table 3.
The following tables show the test and training accuracy on these datasets respectively, as well as results using original features as in Table 3.
For each comparison, we use the same set of model hyperparameters with/without label trick, based on the best performance without label trick.
[Table] [Table]
From Tables 5 and
[Table]
we can see that the degree to which the label trick can improve accuracy is directly related to the training accuracy.
When the training accuracy is near 100%
[Table]
the label trick minimally beneﬁts the model as the label information has already been adequately embedded in the model.
However, when the training accuracy is lower, the label trick can increase both the training and test accuracy signiﬁcantly (once the variance from the dataset divisions is taken into account, please see Appendix A.2).
Moreover, for large-scale graphs it may be less likely that high accuracy or overﬁtting is experienced during training due to the generous number of available samples.
So in such cases, the label trick is more likely to be effective.
A. E FFECT OF R ANDOM D ATASET D IVISION
Due to the random division of the dataset into training, validation and test dataset for Cora-full,
Pubmed, and the regression datasets, the standard deviations across trials are considerably higher than when using ﬁxed splits.
Consequently, for many of the results, the performance gap remains consistent across trials such that the improvement is actually still signiﬁcant.
As an illustration, Table 7 presents the itemized, trial-to-trial results that were averaged to produce the
Cora-full results presented earlier in Table 1.
This reveals that the label trick consistently improves performance over different dataset splits even after careful tuning of the hyperparameters for the baseline model.
This phenomena is similarly present in many of other experimental results.
[Table]
A.
E FFECT OF R EGULARIZATION F ACTOR WITH S ELF - EXCLUDED L INEAR P REDICTORS
In order to demonstrate the effect of the theoretically-derived regularization factor in Theoremand Corollary 1, we apply the regularizer to classiﬁcation experiments involving the deterministic self-excluded linear predictors from Section 4.2.
Figure 2 provides such an example using the Cora dataset and the TWIRLS linear predictor plus regularization as is varied.
Note that if, there is inﬁnite penalty on the label weights which is equivalent to no labels; in contrast if the regularization factor is zero.
[Figure]
[Table]
[Figure]
From Figure 2 we observe that including labels plus the theoretically-motivated regularization (with the right value for α ) can indeed improve performance.
This suggests that even with a classiﬁcation loss (which only loosely aligns with the stated theorem/corollary assumption of an MSE loss), the theory has some practical/predictive value in producing a useful regularization factor.
B P ROOFS B. P ROOF OF T HEOREM
We ﬁrst deﬁne the mask matrix for input labels: M
[Equation]
where α ∈ (0 , 1) is the label rate, and the mask matrix for output is deﬁned as M out
= M tr − M in .
Then the objective is
[Equation]
Next we compute each term separately.
For notational convenience [Equation] we denote the i -th row and j -th column entry of M in by M ij .
For E splits [ (cid:107) M out Y tr [Equation] 2 F ] , we have
[Equation]
For E splits [tr[ Y Ttr M out P M in Y tr W ]] , since M out = M tr − M in , we then have
[Equation]
Let us calculate E splits [ M in P M in ] .
Since M in is a diagonal matrix, we have
[Equation]
[Equation]
where P tr = M tr P M tr and C tr = diag( P tr )
Then we have tr[ E splits
[Equation]
in P [Equation] in ] Y tr [Equation]
[ Y Ttr C
[Equation]
[Equation]
Therefore,
[Equation]
For E splits [ (cid:107) M out P M in Y tr W [Equation] 2 F ] , we have
[Equation]
Let us consider each term separately
[Equation]
(a) To compute tr[ W T Y
Ttr E splits [ M Tin P T M tr P M in ] Y tr W ] , we ﬁrst deﬁne Q tr = diag( P 2 tr ) 12 , and then E splits (cid:104) M Tin P T M tr P M in (cid:105) ij
[Equation]
E splits
[Equation]
Therefore,
[Equation]
(b) To compute tr[ W T Y
Ttr E splits [ M Tin P T M in P M in ] Y tr W ] [Equation] we have that
[Equation]
• For the diagonal entries, where i =
j , we have
[Equation]
and therefore,
[Equation]
• For the off-diagonal entries, where i (cid:54) =
j , we have
[Equation]
and
[Equation]
From (28) and (30), we know that
[Equation]
Unfortunately because C tr P tr + P tr C tr is not necessarily positive semi-deﬁnite
[Equation]
we cannot simplify the proof using the Cholesky decomposition.
So instead we proceed as follows:
[Equation]
Combining the three terms
[Equation]
we get
[Equation]
The overall result L ( W ) is
[Equation]
where C tr = diag( P tr ) and Q tr = diag( P 2 tr ) 12 .
Next we compute (cid:107) Y tr − M tr
( P − C )
Y tr W (cid:107) 2 F .
Since
[Equation]
tr
[Equation]
) we have that
[Equation]
Therefore
[Equation]
[Equation]
Let U =
(diag( P T P ) − C T C )
[Equation]
and Γ = UY tr .
Then we have
[Equation]
B. P ROOF OF C OROLLARY
We have that
[Equation]
which proves the conclusion
[Equation]
B. P ROOF OF T HEOREM
We begin by noting that
[Equation]
And then by Jensen’s inequality we have
[Equation]
Notice based on the nature of cross-entropy [Equation] if 0 < α < [Equation]
we have
CrossEntropy
[Equation]
Z
[Equation]
[Equation]
Therefore,
[Equation]
B. P ROOF OF T HEOREM
Although the label trick splits are drawn randomly using an iid Bernoulli distribution to sample the elements to be added to either D in or D out , we can equivalently decompose this process into two parts.
First, we draw |D out | from a binomial distribution P [ |D out | ]
=
Binom [ m, 1 − α ] , where m is the number of trials and 1 − α is the success probability.
And then we choose the elements to be assigned to D out uniformly over the possible combinations (cid:0) m |D out | (cid:1) .
Given that when |D out | = 0 there is no contribution to the loss, we may therefore reexpress the lefthand side of (12), excluding the limit, as
[Equation]
where it naturally follows that
E
[Equation]
We also have that
[Equation]
and
[Equation]
From these expressions, we then have that
[Equation]
And given the assumption that (cid:12)(cid:12)(cid:80) i ∈D out (cid:96) (cid:0) y i , f GNN [ X , Y in ; W ] i (cid:1)(cid:12)(cid:12) < B < ∞ for some B
[Equation]
we also have
[Equation]
where the bound is independent of α .
Consequently,
[Equation]
such that when combined with the other expressions above [Equation] the conclusion [Equation]
i = (cid:96) (cid:0)
[Equation]
, f
[Equation] [Equation] [Equation] ] i (cid:1) (51) obviously follows.
C E XPERIMENTAL D ETAILS C. T RAINABLE L ABEL P ROPAGATION
In Table 1, we employ trainable label propagation with the same settings as the original label propagation algorithm, including the trade-off term λ in (1) and the number of propagation steps.
In the implementation, we set λ as 0 .
6 and use 50 propagation steps.
We search the best splitting probability α in the range of { 0 . 05 , 0 . 1 , . . .
C. D ETERMINISTIC L ABEL T RICK APPLIED TO GNN S WITH L INEAR L AYERS
We use three models with linear propagation layers and simply choose one speciﬁc set of hyperparameters and run the model on each dataset with or without label trick in Table 3.
The hyperparameters for each model is described as follows.
For TWIRLS, we use 2 propagation steps for the linear propagation layers and λ is set to
(Yang et al., 2021).
For SIGN, we sum up all immediate results instead of concatenating them to simplify the implementation and speed up training.
Our SIGN model also use 5 propagation steps, and we tune the number of MLP layers from 1 or 2 on each dataset.
And for SGC, the number of propagation steps is set to 3 , and there is one extra linear transformation after the propagation steps.
C. E FFECT OF S PLITTING P ROBABILITY
To further present the effect of splitting probability α , in Figure 1, we use α in the range of { 0 . 00625 , 0 . 0125 , 0 . 025 , 0 . 05 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 , 0 .
95 , 0 . 975 , 0 . 9875 , 0 . 99375 } to compare a linear model (similar to a trainable label propagation algorithm with features and labels as input) with a three-layer GCN (Kipf & Welling, 2017).
Each model uses the same set of hyperparameters, except for the number of epochs, since when α is close to zero or one, the model requires more epochs to converge.
For the linear model, λ is set to 0 .
9 and the number of propagation steps is 9 .
The GCN has 256 hidden channels and an activation function of ReLU.
C. T RAINABLE C ORRECT AND S MOOTH
We begin by restating the predictor formulation of trainable Correct and Smooth in (14):
[Equation]
Note that the computation of ˜ Y s and ˜ Y c does not involve any trainable parameters.
Therefore, we can compute them beforehand for each training split.
We now describe our experimental setup used to produce Table 4.
We ﬁrst trained an MLP with exactly the same hyperparameters as in Huang et al.
For each splitting probability α ∈ { 0 . 1 , 0 . 2 , · · · , 0 . 9 } , we generate 10 splits and precompute ˜ Y s and ˜ Y c .
Then for each training epoch, we cycle over the set of ˜ Y s and ˜ Y c .