C OLD B REW : D ISTILLING G RAPH N ODE R EPRESENTATIONS WITH I NCOMPLETE OR M ISSING N EIGHBORHOODS
Anonymous authors
Paper under double-blind review
A BSTRACT
Graph Neural Networks (GNNs) have achieved state of the art performance in node classiﬁcation, regression, and recommendation tasks.
GNNs work well when high-quality and rich connectivity structure is available.
However, this requirement is not satisﬁed in many real world graphs where the node degrees have power-law distributions as many nodes have either fewer or noisy connections.
The extreme case of this situation is a node may have no neighbors at all, called Strict Cold Start (SCS) scenario. This forces the prediction models to rely completely on the node’s input features.
We propose Cold Brew to address the SCS and noisy neighbor setting compared to pointwise and other graph-based models via a distillation approach. We introduce feature-contribution ratio (FCR), a metric to study the viability of using inductive GNNs to solve the SCS problem and to select the best architecture for SCS generalization.
We experimentally show FCR disentangles the contributions of various components of graph datasets and demonstrate the superior performance of Cold Brew on several public benchmarks and proprietary e-commerce datasets. Our codes will be fully open-sourced upon acceptance.
I NTRODUCTION
Graph Neural Networks (GNNs) achieve state-of-the-art results across a wide range of tasks such as graph classiﬁcation, node-classiﬁcation, -regression, link prediction, and recommendation (Wu et al., 2020; Goyal & Ferrara, 2018; Kherad & Bidgoly, 2020; Shaikh et al., 2017; Silva et al., 2010;
Zhang et al., 2019).
GNNs rely on the principle of message passing to aggregate node features across multi-hop neighborhoods in order to learn aggregated representations.
The success of modern GNN models relies on the presence of dense connections and high-quality neighborhoods.
State-of-the-art
GNNs (Kipf & Welling, 2016; Veli ˇ ckovi ´ c et al., 2017; Hamilton et al., 2017; Xu et al., 2018a; Wu et al., 2019; Klicpera et al., 2018) beneﬁt from efﬁcient message-passing frameworks developed to best leverage (deep) neighborhood information to learn node representations.
Other methods like the spectral diffusion (Joachims, 2003; Chaudhuri et al., 2012; Zhang & Rohe, 2018; Wang et al., 2021) and label propagation (Huang et al., 2020a; Zhu, 2005) also require a non-empty and high-quality neighborhood per node to perform message passing.
Even the inductive GNNs (e.g., Hamilton et al.
(2017)) learn a function of the node feature and the node neighborhood, requiring the neighborhood to be present during the inference time.
Most of the large-scale real-world graphs are power-law in nature, with a majority of the nodes having very few connections (Hao et al., 2021;
Ding et al., 2021; Lam et al., 2008; Lu et al., 2020). In Figure 1 we conceptually show the long-tail distribution of such a graph and also empirically show this characteristics on several public datasets.
In many information retrieval and recommender systems applications, there is another ubiquitous challenge: the Strict Cold Start ( SCS ) (Li et al., 2019b; Ding et al., 2021), wherein some “nodes” will have no edges in the graph.
For instance, in most of the public and proprietary datasets we used in the experimentation we have anywhere between 3% to 6% isolated nodes with no neighbors. In these cases, existing GNNs fail to perform due to sparsity or absence of neighborhood.
In this paper, we develop GNN models that can achieve truly inductive capabilities: one can learn effective node embeddings for “orphaned” nodes in a graph.
This capability is important to fully realize the potential of large-scale GNN models on modern, industrialsized datasets with very long tails and nodes with no neighbors in the graph.
To this end,
[Figure]
we adopt the teacher-student knowledge distillation procedure (Yang et al., 2021; Chen et al., 2020b) and propose Cold Brew to distill the knowledge of a GNN teacher into a multilayer perceptron (MLP) student.
The Cold Brew framework boils down to addressing two key questions: (1) how we efﬁciently distill the teacher’s knowledge for the sake of tail and cold-start generalization, and (2) how a student can make use of this knowledge.
We answer these two question by learning a latent node-wise embedding using knowledge distillation, which both avoid “over-smoothness” (Oono & Suzuki, 2020; Li et al., 2018; NT & Maehara, 2019) and could discover the the neighborhoods upon their absence. We discuss everything else in detail in the following sections.
Note that in contrast to traditional knowledge distillation (Hinton et al., 2015), our aim is not to train a simpler student model to perform as well as the more complex teacher.
Instead, we aim to train a student model that is better than the teacher, and generalizes to samples in the same graph where the teacher will be ineffective.
To help select the cold-start-friendly model architectures, we develop a metric called Feature-Contribution Ratio (FCR) that disentangles the graph data into node features and the neighborhood structure.
We then build submodules for each part and search for the best architecture.
Besides selecting the optimal model architecture, we leverage FCR to provide a measure of importance of the node feature compared to the neighborhood structure, quantifying the difﬁculty of learning truly inductive GNNs.
We summarize our key contributions as follows:
• To generalize better to tail and SCS nodes, we design the Cold Brew framework to distill a GNN teacher into an MLP student.
We improve over GNN models by adding the node-wise Structural Embedding (SE) to the Cold Brew’s teacher GNN to strengthen the expressiveness of the teacher GNN model.
We designed a novel mechanism for the MLP student to discover the “latent/virtual neighborhoods” when they are missing, and perform message passing as its teacher model does.
• We propose Feature-Contribution Ratio (FCR): a new metric for graph datasets that quantiﬁes the contribution of node features w.r.t.
the adjacency structure in the dataset for a speciﬁc downstream task.
FCR indicates the difﬁculty in generalizing to tail and cold-start nodes in a given dataset.
We also leverage FCR as a principled “screen process” to select the best model architecture for both the GNN teacher and the MLP student in Cold Brew.
• Through extensive experiments on several public datasets as well as a proprietary ecommerce graph dataset, we validate the effectiveness of Cold Brew in cold-start generalization. We further uncover the contribution ratio of node features in node prediction tasks with FCR.
1. P ROBLEM S ETUP
GNNs effectively learn the representations of two components in graph data: they process node features by distributed node-wise transformations and process adjacency structure by localized neighborhood aggregations.
For the ﬁrst component, GNNs apply shared feature transformations to all nodes regardless of the neighborhoods, while for the second component, GNNs use permutationinvariant aggregators to collect neighborhood information.
We take the node classiﬁcation problem in this paper for ease of exposition, and the proposed method can be trivially adapted to semi-supervised/unsupervised settings.
We denote the graph data of interest by G with node set V , |V| = N .
Each node possesses a d in − dimensional feature and a d out − dimensional label (either d out classes or a continuous vector in the case of regression).
Let
X 0 ∈ R N × d in and Y ∈ R N × d out be the matrices of node features and labels, respectively.
Let N i be the neighborhood of node i ∈ [ N ] .
As is common with modern, large-scale graphs, |N i | is small for several nodes in the graph, which we refer to as tail nodes.
For a number of nodes, |N i | = 0 , and we refer to these nodes as cold-start nodes.
A traditional GNN learns representations for the i th node at the l th layer as a function of the node representation itself and its neighborhoods’ representations, at the ( l − 1) th layer:
[Equation]
where f ( · ) is a general function that applies node-wise transformation on node x l − 1 i and aggregates information of its neighborhood { x l − 1 j } j ∈N i to obtain the ﬁnal node representation.
Given i ’s input features x 0 i and its neighborhood N i , one can use (1) to obtain its representation and predict y i , making these models inductive.
We are interested in improving the performance of these GNNs on a set of tail and cold-start nodes, where N i for node i is either unreliable 1 or absent.
In these cases, applying (1) will yield a suboptimal node representation, since { x l − 1 j } j ∈N i will be unreliable or empty at inference time.
R ELATED W ORK
GNNs learn by aggregating neighborhood information to learn node representations (Kipf & Welling, 2016; Veli ˇ ckovi ´ c et al., 2017; Hamilton et al., 2017; Xu et al., 2018a; Wu et al., 2019; Klicpera et al., 2018).
Inductive variants of GNNs such as GraphSAGE (Hamilton et al., 2017) require initial node features as well as the neighborhood information of each node to learn the representation.
Most works on improving GNNs have focused on learning better aggregation functions, and methods that can work when the neighborhood is absent or noisy have not been sufﬁciently exploited, except two very recent or concurrent works (Hu et al., 2021; Zhang et al., 2021).
In the context of cold start, (Hao et al., 2021) and (Ding et al., 2021) employ a transfer learning approach.
(Yang et al., 2021) proposes a knowledge distillation for GNN, while (Chen et al., 2020b) proposes a self-distillation approach.
In all the above cases, the models need full knowledge of the neighbors of the cold-start nodes in question and do not address the case of noisy or missing neighborhoods.
Another possible solution is to directly train an MLP that only take node features.
(Hu et al., 2021) proposes to learn graph embeddings with only node-wise MLP, while using contrastive loss to regularize the graph structure.
Some previous works have studied the relation between node feature similarity and edge connections and studied their inﬂuence on the selection of appropriate graph models.
(Pei et al., 2020) proposed the homophily metric that categorizes the graphs into assortative and disassortative classes.
(Wang et al., 2021) dissected the feature propagation steps of linear GCNs from a perspective of continuous graph diffusion and analyzed why linear GCNs fail to beneﬁt from more propagation steps.
(Liu et al., 2020a) further studied the inﬂuence of homophily on model selection and proposed a non-local
GNN.
S TRICT C OLD S TART G ENERALIZATION
We now address the problem of generalization to the tail and cold-start nodes, where the neighborhood information is missing/noisy (Section 1).
A naive baseline is to train an MLP to map node features to labels.
However, such method disregards all graph information, and we show via our FeatureContribution Ratio and other experimental results that for most assortative graph datasets, the node-wise MLP approach is suboptimal.
The key idea of our framework is the following: the GNN maps node features into a d -dimensional embedding space, and since the number of nodes N is usually much bigger than the embedding dimensionality d , we end up with an overcomplete set for [-tail and cold-start nodes. We use a teacher GNN model to x l +
 i-]this space using the embeddings as the basis.
This implies the possibility that any node representation can be cast as a linear combination of
K (cid:28) N existing node representations.
Our aim will be to train a student model that can accurately discover the combination of the best K existing node embeddings of a target isolated node.
We call
[Figure]
this procedure latent/virtual neighborhood discovery , which is equivalent to using MLPs to “mimic” the node representations learned by the teacher GNN. We adopt the knowledge distillation procedure (Yang et al., 2021; Chen et al., 2020b) to improve the quality of the learned embeddings for tail and cold-start nodes. We use a teacher GNN model to embed the nodes onto a low-dimensional manifold by utilizing the graph structure.
Then, the job of the student is to learn a mapping from the node features to this manifold without knowledge of the graph that the teacher has.
We further aim to let the student model generalize to SCS cases where the teacher model fails, beyond just mimicking the teacher as standard knowledge distillation does.
3. T HE T EACHER M ODEL OF C OLD B REW : S TRUCTURAL E MBEDDING GNN
Consider a graph G .
For a Graph Convolutional Network with L layers, the l -th layer transformation can be written as 2 : X ( l +1) =
σ ( ˜ A X ( l ) W ( l ) ) , where ˜ A is the normalized adjacency matrix, ˜
A =
D − 1 / 2 AD − 1 / 2 , D is the diagonal degree matrix and A is the adjacency matrix. X ( l ) ∈ R
N × d 1 is the node representations in the l -th layer,
W ( l ) ∈ R d 1 ×
d 2 is the feature transformation matrix, where the values of d 1 /d 2 depend on layer l : ( d 1 , d 2 ) =
( d in , d hidden ) for l = 0 , ( d hidden , d hidden ) for 1 ≤ l ≤ L − 2 , and ( d hidden , n classes ) for l =
L − 1 .
σ ( · ) = is the nonlinear functions applied to each layer, (e.g. ReLU).
Norm ( · ) refers to an optional batch or layer normalization.
GNNs typically suffer from oversmoothing (Oono & Suzuki, 2020; Li et al., 2018; NT & Maehara, 2019), making all the node representations similar to each other.
This is harmful not only for the
GNN itself, but also to the student model that looks to mimic the GNN. Inspired by the positional encoding in Transformers (Vaswani et al., 2017), we train the teacher GNN to additionally learn a set of node embeddings which we term the Structural Embedding (SE) .
SE learns to incorporate additional information (such as node labels in the case of semi supervised learning) through gradient backpropagation.
The existence of SE avoids the oversmoothing issue in GNNs: the transformations applied to different nodes are no longer the same for each nodes, since the SE of each node is different, which is part of the feature transformation function.
Note that this could be of independent interest to practitioners who train and study GNNs.
Speciﬁcally, for each layer l, 0 ≤ l ≤ L − 1 , the Structural Embedding is a learnable matrix E ( l ) , and the SE-GNN layer forward pass can be written as:
[Equation]
Remark 1: Note that SE is not the same as the bias term in traditional feature transformation
X ( l +1) =
σ (cid:16) ˜
A (cid:0) X ( l ) W ( l ) + b ( l ) (cid:1)(cid:17) , where for the bias b ∈ R N × d 2 the rows are copied/shared across all nodes.
In contrast, we have a different structural embedding for every node.
Remark 2: SE is also unlike traditional label propagation (LP) (Iscen et al., 2019; Wang & Leskovec, 2020; Huang et al., 2020b). LP encodes label information through iterating E ( t +1)
= (1 − α )
G + α ˜
A E ( t ) , where G is a one-hot encoding of ground truth for training node classes and zeros for test nodes, and 0 < α < 1 is the portion of mixture at each iteration.
SE-GNN enables node i to learn to encode the self and neighbors’ label information 3 into its own node embedding through ˜ A .
We use the Graph Convolutional Networks (Kipf & Welling, 2016), combined with other building blocks proposed in recent literature including: (1) initial/dense/jumping connections; and (2) batch/pair/node/group normalization, as the backbone of Cold Brew’s teacher
GNN.
More details are described in Appendix A. We also apply a regularization term to the loss function, yielding the following loss function:
[Equation]
where X ( L ) train is the model’s embedding at the L -th layer, which is exactly the model’s output, CE ( X ( L ) train , Y train ) is the Cross Entropy between the model output X ( L ) train and the ground truth Y on the training set, and η is the coefﬁcient (grid-searched for different datasets in practice).
The Cross-Entropy loss can be replaced by any other appropriate loss depending on the task.
3. T HE S TUDENT MLP M ODEL OF C OLD B REW
We design the student to be composed of two MLP modules. Given a target node, the ﬁrst MLP module imitates the node embeddings generated by the GNN teacher. Next, given any node, we ﬁnd a set of virtual neighbors of that node from the graph. Finally, the second MLP attends to both the target node and the virtual neighborhood and transforms them into the embeddings of interest.
Suppose we would like to obtain the embedding of an potentially isolated target node i , given only its feature x i .
From the teacher GNN, at each layer l , we have access to two sets of node embeddings: X ( l ) W ( l ) and E ( l ) .
Denote ¯ E as the desired embeddings that the teacher GNN pass over to the student MLP.
We offer two options for ¯ E : it can be the ﬁnal output of the teacher GNN, in this case, ¯ E ∈ R N × d out :=
X ( L ) , or optionally, it can be the concatenation of all intermediate results of the teacher GNN, similar to (Romero et al., 2014):
¯ E ∈ R N × d hidden ∗ ( L − 1)+ d out ) :=
X ( L ) (cid:83) L − 1 l =
( E ( l ) + X ( l ) W ( l ) ) , where (cid:83) is the concatenation of matrices at the feature dimension (second dimension).
¯ E acts as the target for the ﬁrst MLP module and the input to the second MLP module.
The ﬁrst MLP learns a mapping from the input node features X (0) to ¯ E , that is ξ 1 : x (0) i
→ e i for node i , e i = ¯ E
[ i, :] .
Then, we discover the virtual neighborhood by applying an attention-based aggregation of the existing embeddings in the graph, before linearly combining them:
[Equation]
where Θ K ( · ) is the top- K hard thresholding operator: for z ∈ R 1 × N : [Θ K ( z )]
j = z j if z j is among the top- K largest elements of z , and Θ K ( z ) j = −∞ otherwise.
Finally, the second MLP learns a mapping ξ 2 : [ x i , e i ] → y i , where y i = Y [ i, :] is the ground truth for node i . Equation (4) ﬁrst selects K nodes from the N nodes that the teacher GNN was trained on via the hard thresholding operator.
˜
e j is then a linear combination of K node 4 embeddings. Thus, every sample whether or not seen previously while training the GNN can be represented as a linear combination of these representations. The MLP ξ 2 ( · ) maps this representation to the ﬁnal target of interest. Thus, we decompose every node embedding as a linear combination of an (overcomplete) basis.
The training of ξ 1 ( · ) is by minimizing the mean squared error over the non-isolated node in the graph (mimicing the teacher’s embeddings), and the training of ξ 2 ( · ) is by minimizing the cross entropy (for node classiﬁcation task) or mean squared error (for node regression task) on the training split of the tail and isolated part of the graph.
An illustration of SE-MLP’s inference procedure for the isolated nodes is shown in Fig.
When the number of nodes is large, the ranking procedure involved in Θ K ( · ) can be pre-computed after training the ﬁrst part and before training the second part.
3. M ODEL I NTERPRETATION F ROM A L ABEL S MOOTHING P ERSPECTIVE
We cite Theorem 1 in (Wang & Leskovec, 2020):
Suppose that the latent ground-truth mapping from node features to node labels is differentiable and L-Lipschitz.
If the edge weights a ij approximately smooth x i over its immediate neighbors with error (cid:15) i , i.e., x i = 1 d ii Σ
j ∈N a ij x j + (cid:15) i , then the a ij also approximately smooth y i to bound within error |
y i − 1 d ii Σ j ∈N i a ij y j | ≤ L || (cid:15)
||
2 + o (max j ∈N
i ( || x j − x i || 2 )) , where o ( · ) denotes a higher order inﬁnitesimal.
This theorem indicates that the errors of the label predictions are determined by the difference of the features after neighborhood aggregation: if (cid:15) i is large, then the error in the label prediction is also large, and vice versa.
However, with structural embedding, each node i also learns an independent embedding ¯ E [: , i ] during the aggregation, which changes 1 d ii Σ j ∈N a ij x j + (cid:15) i into 1 d ii Σ j ∈N a ij x j + ¯ E [: , i ]+ (cid:15) i . Deduced from this theorem, the structural embedding ¯ E is important for the teacher model: it allows higher ﬂexibility and expressiveness in learning the residual difference between nodes, and hence the error (cid:15) i can be are lowered if ¯ E is properly learned.
From this theorem, one can also see the necessity of introducing neighborhood aggregations like the Cold Brew’s student model. If one directly applies MLP models without neighborhood aggregation, the (cid:15) i turns out to be non-negligible, leading to higher losses in the label predictions. However, Cold
Brew introduced the neighborhood aggregation mechanism, so that the second part of the student MLP takes over the aggregation of neighborhood generated by the ﬁrst MLP. Therefore, Cold Brew eliminates the above residual error even in the absence of the actual neighborhood.
M ODEL S ELECTION A ND G RAPH C OMPONENT D ISENTANGLEMENT W ITH F EATURE -C ONTRIBUTION R ATIO
We now discuss Feature Contribution Ratio (FCR): a metric to quantify the difﬁculty of learning representations under the truly inductive cold start case, and a hyperparameter optimization approach to select the best suitable model architecture that helps tail and cold start generalization.
As conceptually illustrated in Figure 2, there are four atomic components contributing to the learned embedding of node i in the graph: 1.
the label of i ( self-label ); 2.
the label of neighbors of i ( neighborlabels ); 3.
the features of i ( self-feature ); 4.
the features of neighbors of i ( neighbor-features ).
To quantize the SCS generalization difﬁculty, we divide these four components into two submodules to disentangle the contributions of the node features with respect to the adjacency structure of the graph dataset, and quantize it based on the assumption that the SCS generalization difﬁculty is propotional to the contribution ratio of the node features .
We posit that a submodule that learns accurate node representations must include the node’s (self) label, so that training can be performed via backpropagation.
What remains is to use the label with other atomic components to construct two specialized models that each make use of only the node features or the adjacency structure.
For the ﬁrst submodule, we build an MLP that maps the selffeatures to self-labels, ignoring any neighborhood information present in the dataset.
For the second submodule, we adopt a Label Propagation (LP) method (Huang et al., 2020a) 5 to learn representations from self- and neighbor-labels.
This model ignores any node feature information.
With the above two submodules, we introduce the Feature-Contribution Ratio (FCR) that characterizes the relative importance of the node features and the graph structure.
Speciﬁcally, for graph dataset
G , we deﬁne the contribution of a submodule to be the residual performance of the submodule compared to a full-ﬂedged GNN (e.g., Equation (1)) using both the node feature as well as the adjacency structure.
Denote z
MLP , z LP , and z GNN as the performance of the MLP submodule, LP submodule, and the full GNN on the test set, respectively.
If z MLP (cid:28) z GNN , then FCR ( G ) is small and the graph structure is important, and noisy or missing neighborhood information will hurt model performance.
Based on this intuition, we build SCR as:
[Equation]
Interpreting FCR values.
For a particular graph G , if 0% ≤ FCR ( G ) < 50% , it means z GNN > z LP > z MLP , and the neighborhood information in G is mainly responsible for the GNN achieving good performance.
If 50% ≤ FCR ( G ) < 100% , then z GNN > z MLP > z LP , the node features contribute more to the GNN’s performance.
If FCR ( G ) ≥ 100% , then z MLP > z GNN > z LP , and the node aggregation in GNNs can actually lead to reduced performance compared to pointwise models.
This case usually happens for some disassortative graphs (the majority of neighborhoods hold different labels than the center node), e.g. as observed by (Liu et al., 2020a) .
Integrate FCR as a tool to design teacher and student models.
For some graph datasets and models, the SCS generalization can be challenging without neighborhood information (i.e, z GNN > z LP > z MLP ).
We hence consider FCR as a principled “screening process” to select model architectures for both teacher and student, that own the best inductive bias for SCS generalization.
To achieve this, during the computation of FCR, we perform exhaustive grid search of the architectures (residual connection types, normalization, hidden layers, etc.) for the MLP, LP, and GNN modules, and pick the best-performing variant.
Detailed deﬁnition of the search space can be found in
Appendix A. We treat this grid search procedure as a special case of architecture selection and hyperparameter optimization for Cold Brew.
We observe that FCR is able to identify the GNN and MLP architectures that are particularly friendly for SCS generalization, improving our method design.
In experiments, we observe that different model conﬁgurations are favored by different datasets, and we use the found optimal teacher GNN and student MLP architectures to perform Cold Brew.
More detailed discussions will be presented in section 5.
E XPERIMENTS AND D ISCUSSION
In this section, we ﬁrst evaluate the FCR for several commonly used graph datasets to ascertain how well GNNs trained on them can generalize to tail and cold-start nodes.
We also compare it to the the graph homophiliy metric β proposed in (Pei et al., 2020).
Next, we apply Cold Brew to these datasets and compare its generalization ability to baseline graph-based as well as purely pointwise models on these datasets.
We also show results on proprietary industry datasets.
5. D ATASETS AND S PLITS
We ﬁrst detail the datasets used in this paper.
We perform studies on several open-source datasets, together with four proprietary datasets.
The proprietary E-commerce datasets, E-comm 1/2/3/4, refers to graphs subsampled from anonymized logs of an e-commerce store.
They are sampled so as to not reﬂect the actual raw trafﬁc distributions, and results are provided with respect to a baseline model for these datasets.
The different datasets refer to different product subsets, and the labels indicate product categories that we wish to determine.
Node features are text embeddings obtained from a ﬁne-tuned BERT model.
We show FCR values for the public datasets and exhaustively evaluate the proposed Cold Brew as well as several baselines for ﬁve public datasets and the proprietary datasets.
The statistics of the datasets are summarized in Table 1.
We create speciﬁc training and test splits of the datasets in order to speciﬁcally study the generalization ability of Cold Brew to tail and cold-start nodes.
In the following tables, the head data corresponds to the top 10% highest degree nodes in the graph, and the subgraph that they induce.
The tail data corresponds to the 10% nodes in the graph with lowest (non-zero) degree, and the subgraph that they induce.
All the zero degree nodes are in the isolation data.
For datasets that don’t have isolated nodes, we manually remove edges from 10% lowest degree nodes.
The Overall data refers to the training/test splits without distinguishing head/tail/isolation.
[Table]
5. FCR E VALUATION
In Table 2, the top half presented the FCR results together with the homophily metric β in (Pei et al., 2020) (Equation 6).
The below half uncovered the different prediction quality for the head and the tail nodes. As can be seen from the table, as an indicator of the tail generalization difﬁculty, FCR differs among datasets, and is negatively correlated with the homophily metric (with pearson correlation coefﬁcient -0.76). The high absolute correlation value and its negative sign indicate that the more similar the nodes are to their neighborhoods, the more difﬁcult it is to generalize with MLP based models. The evaluations on more datasets (including the datasets where FCR > 100% ) are presented in Appendix B.
[Table]
the number of v ’s direct neighbors that have the same labels as v the number of v
[Table]
directly connected neighbors
[Table]
5. E XPERIMENTAL R ESULTS ON T AIL G ENERALIZATION W ITH C OLD -B REW
In Table 3, we present the performances of the Cold Brew together with baselines on the tail and the isolation splits, across several different datasets.
All models in the table are trained on the overall training subset, and are evaluated on the hold-out test subsets of tail and isolation splits (discussed in section 5.1).
The GCN refers to the the best conﬁguration found through FCR-guided grid search (check Appendix A for details), without Structural Embedding. Correspondingly, GCN + SE refers to the best FCR-guided conﬁguration with Structural Embedding, which is the default teacher model of
Cold Brew. GraphSAGE refers to (Hamilton et al., 2017), Simple MLP refers to a simple node-wise
MLP that has two hidden layers with 128 hidden dimensions, and GraphMLP refers to (Hu et al., 2021).
The results for E-commerce datasets are presented as relative improvements to the baseline (each value is the difference w.r.t.
the value of the GCN 2 layers on same dataset of the same split), and we do not disclose the absolute numbers due to proprietary reasons.
As shown in Table 3, Cold Brew’s student MLP improves accuracy on isolated nodes by up to +11% on the E-commerce datasets and +2.4% on open source datasets.
Cold Brew’s student model handles isolated nodes much better, and it teacher GNN also achieves better performance in the tail split, compared to all other models.
Especially, when compared with GraphMLP, Cold Brew’s student MLP shows consistently better performance in most datasets. This can be explained from their different mechanisms: GraphMLP encodes graph knowledge implicitly in the learned weights, while Cold Brew explicitly attends to neighborhoods, even when it is absent.
More detailed comparisons can be found in Appendix B. [Table] [Table]
[Table] [Table] We also compared the link prediction tasks to justify the performance under recommendation system settings, and the results are shown in Table 4.
The comparison is done over the isolation splits, where the ground truth edges are manually removed, and the models are asked to recover these edges. The baseline model shown in tables are TailGCN (Vetter et al., 1991), Meta-Tail2Vec (Liu et al., 2020b). A comparison over these models on the node classiﬁcation on the isolation split is provided in Table 5. As observed from the table, cold brew outperformed TailGCN and Meta-Tail2Vec on the isolation split, since both TailGCN and Meta-Tail2Vec explicitely are not zero-shot methods and require explicit neighborhood nodes, hence their performance degrade when the neighborhood is empty and padded by zeros vectors.
There are other interesting observations. First, if the FCR is high (such as Pubmed), then for this dataset the MLP-type models tend to outperform GNN-type models in all splits.
Regardless of FCR, for almost all datasets, the MLP-type models outperform the GNN-type models on the isolation split, and a few on the tail split, while the GNN-type models are superior in other splits.
Second, the proposed structural embeddings imply a tantalizing potential to alleviate the over-smoothness (Oono & Suzuki, 2020; Li et al., 2018; NT & Maehara, 2019) and bottleneck (Alon & Yahav, 2020)
issues observed in deep GCN models.
As shown in table Table 6, Cold Brew’s GCN (GCN + SE) signiﬁcantly outperformed the traditional GCN on 64 layers: the former has 34% test accuracy higher on Cora, 23% higher on Citeseer and similar on others.
Indeed, the improvement over isolation and tail splits (especially the isolation split) comes with a cost: we observed a performance drop for the student MLP model on the head and several other datasets’ tail splits, compared with the naive GCN model. The full performance on other splits are listed in the appendix as a reference.
However, the cold-brew speciﬁcally targets at the challenging strict cold start issues, as a new compelling alternative for in these cases. Meanwhile in the non-cold-start cases, the traditional GCN models can still be used to obtain good performance.
Note that, even on the head splits, the proposed GNN teacher model of Cold Brew still outperformed traditional GNN models. We hence consider as promising future work to adaptively switch between using Cold Brew teacher and student models, based on the current node connectivity degree and tailedness. [Table]
C ONCLUSION
In this paper [Table] we studied the problem of generalizing GNNs to the tail and strict cold start nodes, whose neighborhood information is noisy or missing.
We proposed a teacher-student knowledge distillation procedure to better generalize to the isolated nodes.
We added an independent set of structural embeddings in GNN layers to to alleviate node over-smoothness [Table]
and also proposed a virtual neighbor discovery step for the student model to attend to estimated neighborhood nodes.
We additionally established FCR, a metric for quantifying the difﬁculty of truly inductive representation learning, to optimize our method design.
Experiments demonstrated the consistently superior performance of our proposed framework, on several common benchmarks as well
R EPRODUCIBILITY C HECKLIST
To ensure reproducibility, we use the Machine Learning Reproducibility Checklist v2.0, Apr.2020 (Pineau et al., 2021).
• For all models and algorithms presented, – A clear description of the mathematical settings, algorithm, and/or model.
We clearly describe all of the settings, formulations, and algorithms in Section 5.
– A clear explanation of any assumptions.
We do not make assumptions.
– An analysis of the complexity (time, space, sample size) of any algorithm.
We do not make the analysis.
• For all datasets used, check if you include: – The relevant statistics, such as number of examples.
We use widely adopted public graph datasets in Section 5.
We provide all the related statistics in Appendix 5.
– The details of train/validation/test splits.
We give this information in section 5.
– For new data collected,a complete description of the data collection process, such as instructions to annotators and methods for quality control.
We do not collect or release new datasets.
• For all reported experimental results , check if you include: – The range of hyper-parameters considered, method to select the best hyperparameter conﬁguration, and speciﬁcation of all hyper-parameters used to generate results.
We provide all details of the hyper-parameter tuning in Section 5.
– The exact number of training and evaluation runs.
Ten independent repetitions are conducted for each experiments.
– A clear deﬁnition of the speciﬁc measure or statistics used to report results.
We use the classiﬁcation accuracy on test-set, which is deﬁned in Section 5.
– A description of results with central tendency (e.g. mean) & variation (e.g. error bars).
We report mean for all experiments, as indicated in Section 5.
– The average runtime for each result, or estimated energy cost.
We do not report the running time or energy cost.
A S EARCH S PACE D ETAILS
In the computation of FCR, we include a search space of model hyperparameters for GNN, MLP and
LP, in order to ﬁnd the best suitable conﬁgurations for distillation.
For GNN model, we take GCN as a backbone and performed grid search over the number of hidden layers, whether it has structural embedding or not, the type of residual connection, and the type of normalization.
For the number of hidden layers, we considered 2, 4, 8, 16, 32, and 64.
For the types of residual connections, we include: (1) connecting to the last layer (Li et al., 2019a; 2018), (2) initial connection connecting to the initial layer (Chen et al., 2020a; Klicpera et al., 2018; Zhang et al., 2020), (3) dense connection connecting to all the preceding layers (Li et al., 2019a; 2018; 2020;
Luan et al., 2019), and (4) jumping connection combining all all the preceding layers only at the ﬁnal graph convolutional layer (Xu et al., 2018b; Liu et al., 2020b).
For the types of normalizations, we grid search over: batch normalization (BatchNorm) (Ioffe & Szegedy, 2015), pair normalization (PairNorm) (Zhao & Akoglu, 2019) node normalization Zhou
), mean normalization Yang ), and differentiable group normalization (GroupNorm) Zhou types of graph dropout methods,
we Dropout
For the architecture design for Cold Brew’s MLP, we conducted hyperparameter search over the number of hidden layers, the existence of residual connection, the hidden dimensions, and the optimizers. [Table] [Table]
The number of hidden layers is searched over (2, 8, 16, 32).
The number of hidden dimensions is searched over (128, 256).
The optimizer is searched over (Adam(lr=0.001)
Adam(lr=0.005),
Adam(lr=0.02), SGD(lr=0.005))
For the Label Propagation, we conducted hyperparameter search over number of propagations, propagation matrix type, and the mixing coefﬁcient α (Huang et al., 2020a).
The number of propagations is searched over (10, 20, 50, 100, 200).
The propagation matrix type is searched over adjacency matrix or normalized Laplacian matrix.
The mixing coefﬁcient α is searched over (0.01, 0.1, 0.5, 0.9, 0.99).
The best GCN, MLP, and LP conﬁgurations are reported in Tables 7, 8, and 9, respectively.
[Table] [Figure]
The performance evaluations on all splits are listed in Table 10. The FCR evaluation on more datasets are presented in Fig 11. C V ISUALIZING THE LEARNED EMBEDDINGS
Figure 3 visualizes the last-layer embeddings of different models after t-SNE dimensionality reduction. In the ﬁgure, colors denotes node labels and all nodes are marked as dots, with isolation subset nodes additionally marked with ‘ x ’s and the tail subset additionally marked with triangles. Although the GCN model did a decent job in separating different classes, a signiﬁcant portion of the tail and isolation nodes fall into wrong class clusters. Cold Brew’s MLP is much better discriminative in the tail and isolation splits.
[Table]