B OOST THEN C ONVOLVE : G RADIENT B OOSTING M EETS G RAPH N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Graph neural networks (GNNs) are powerful models that have been successful in various graph representation learning tasks.
Whereas gradient boosted decision trees (GBDT) often outperform other machine learning methods when faced with heterogeneous tabular data.
But what approach should be used for graphs with tabular node features?
Previous GNN models have mostly focused on networks with homogeneous sparse features and, as we show, are suboptimal in the heterogeneous setting.
In this work, we propose a novel architecture that trains GBDT and GNN jointly to get the best of both worlds: the GBDT model deals with heterogeneous features, while GNN accounts for the graph structure.
Our model beneﬁts from end-to-end optimization by allowing new trees to ﬁt the gradient updates of GNN.
With an extensive experimental comparison to the leading GBDT and GNN models, we demonstrate a signiﬁcant increase in performance on a variety of graphs with tabular features.
I NTRODUCTION
Graph neural networks (GNNs) have shown great success in learning on graph-structured data with various applications in molecular design (Stokes et al., 2020), computer vision (Casas et al., 2019), combinatorial optimization (Mazyavkina et al., 2020), and recommender systems (Sun et al., 2020).
The main driving force for progress is the existence of canonical GNN architecture that efﬁciently encodes the original input data into expressive representations, thereby achieving high-quality results on new datasets and tasks.
Recent research has mostly focused on GNNs with sparse data that represent either homogeneous node embeddings (e.g., one-hot encoded graph statistics) or bag-of-words representations.
Yet tabular data with detailed information and rich semantics among nodes in the graph are more natural for many situations and abundant in real-world AI (Xiao et al., 2019).
For example, in a social network, each person has socio-demographic characteristics (e.g., age, gender, date of graduation), which largely vary in data type, scale, and missing values.
GNNs for graphs with tabular data remain unexplored, with gradient boosted decision trees (GBDTs) largely dominating in applications with such heterogeneous data (Bent´ejac et al., 2020).
GBDTs are so successful for tabular data because they possess certain properties: (i) they efﬁciently learn decision space with hyperplane-like boundaries that are common in tabular data; (ii) they are well-suited for working with variables of high cardinality, features with missing values and of different scale; (iii) they provide qualitative interpretation for decision trees (e.g., by computing decrease in node impurity for every feature) or for ensembles via post-hoc analysis stage (Kaur et al., 2020); (iv) in practical applications they mostly converge faster even for large amounts of data.
In contrast, a crucial feature of GNNs is that they take into account both the neighborhood information of the nodes and the node features to make a prediction, unlike GBDTs that require additional preprocessing analysis to provide the algorithm with graph summary (e.g., through unsupervised graph embeddings (Hu et al., 2020a)).
Moreover, it has been shown theoretically that message-passing
GNNs can compute any function on its graph input that is computable by a Turing machine, i.e.,
GNN is known to be the only learning architecture that possesses universality properties on graphs (approximation (Keriven & Peyr´e, 2019; Maron et al., 2019) and computability (Loukas, 2020)).
Furthermore, gradient-based learning of neural networks can have numerous advantages over the treebased approach: (i) relational inductive bias imposed in GNNs alleviates the need to manually engineer features that capture the topology of the network (Battaglia et al., 2018); (ii) the end-to-end nature of training neural networks allows multi-stage (Fey et al., 2020) or multi-component (Wang et al., 2020) integration of GNNs in application-dependent solutions; (iii) pretraining representations with graph networks enriches transfer learning for many valuable tasks such as unsupervised domain adaptation (Wu et al., 2020), self-supervised learning (Hu et al., 2020b), and active learning regimes (Satorras &
Estrach, 2018).
Undoubtedly, there are major beneﬁts in both GBDT and GNN methods.
Is it possible to take advantages of both worlds?
All previous approaches (Arik & Pﬁster, 2020; Popov et al., 2019; Badirli et al., 2020) that attempt to combine gradient boosting and neural networks are computationally heavy, do not consider graph-structured data, and suffer from the lack of relational bias imposed in
GNN architectures.
Other limitations of related work can be found in Appendix A. To the best of our knowledge, the current work is the ﬁrst to explore using GBDT models for graph-structured data.
In this paper, we propose a novel learning architecture for graphs with tabular data, BGNN , that combines GBDT’s learning on tabular node features with GNN that reﬁnes predictions utilizing the graph’s topology.
This allows BGNN to inherit the advantages of gradient boosting methods (heterogeneous learning and interpretability) and graph networks (representation learning and end-toend training).
Overall, our contributions are the following:
(1) We design a novel generic architecture that combines GBDT and GNN into a unique pipeline.
To the best of our knowledge, this is the ﬁrst work that systematically studies the application of GBDTs to graph-structured data.
(2) We overcome the challenge of end-to-end training of GBDTs by iteratively adding new trees that ﬁt the gradient updates of GNN.
This allows us to backpropagate the error signal from the topology of the network to GBDT.
(3) We perform an extensive evaluation of our approach against strong baselines in node prediction tasks.
Our results consistently demonstrate signiﬁcant performance improvements on heterogeneous node regression and node classiﬁcation tasks over a variety of real-world graphs with tabular data.
(4) We show that our approach is also more efﬁcient than the state-of-the-art GNN models due to much faster loss convergence during training.
Furthermore, learned representations exhibit discernible structure in the latent space, which further demonstrates the expressivity of our approach.
B ACKGROUND
Let G =
( V, E ) be a graph with nodes having features and target labels.
In node prediction tasks (classiﬁcation or regression), some target labels are known and the goal is to predict the remaining ones.
Throughout the text, by lowercase variables x v ( v ∈ V ) or x we denote features of individual nodes, and X denotes the matrix of all features for v ∈ V .
Individual target labels are denoted by y v , while Y all and Y are the vectors of all and only training labels, respectively.
Graph Neural Networks (GNNs) use both the network’s connectivity and the node features to learn latent representations for all nodes v ∈ V .
Many popular GNNs use a neighborhood aggregation approach, also called the message-passing mechanism, where the representation of a node v is updated by applying a non-linear aggregation function of v ’s neighbors representation (Fey & Lenssen, 2019).
Formally, GNN is a differentiable, permutation-invariant function g θ : ( G, X ) (cid:55)→ Y all .
Similar to traditional neural networks, GNNs are composed of multiple layers, each representing a non-linear message-passing function:
[Equation]
where x tv is the representation of node v at layer t , and COMBINE t and AGGREGATE t are (parametric) functions that aggregate representations from the local neighborhood of a node.
Then, the
GNN mapping g θ includes multiple layers of aggregation (1).
Parameters of GNN model θ are optimized with gradient descent by minimizing an empirical loss function L ( Y, g θ ( G, X )) .
Gradient Boosted Decision Trees (GBDT) is a well-known and widely used algorithm that is deﬁned on non-graph tabular data (Friedman, 2001) and is particularly successful for tasks containing heterogeneous features and noisy data.
The core idea of gradient boosting is to construct a strong model by iteratively adding weak ones (usually decision trees).
Formally, at each iteration t of the gradient boosting algorithm, the model f ( x ) is updated in an additive manner:
[Equation]
where f t − 1 is a model constructed at the previous iteration, h t is a weak learner that is chosen from some family of functions H , and (cid:15) is a learning rate.
The weak learner h t ∈ H is chosen to approximate the negative gradient of the loss function L w.r.t.
the current model’s predictions:
[Equation]
The gradient w.r.t.
the current predictions indicates how these predictions should be changed in order to improve the loss function.
Informally, gradient boosting can be thought of as performing gradient descent in functional space.
The set of weak learners H is usually formed by shallow decision trees.
Decision trees are built by a recursive partition of the feature space into disjoint regions called leaves.
This partition is usually constructed in a greedy manner to minimize the loss function (3).
Each leaf R j of the tree is assigned to a value a j , which is an estimate of the response y in the corresponding region.
In our case a j is equal to the average negative gradient value in the leaf R j .
To sum up, we can write h ( x ) =
(cid:80) j a j 1 { x ∈ R j } .
GBDT MEETS GNN
[Table]
𝐺, 𝑋′
Loss 𝐿 𝑔 𝜃 𝐺,𝑋 ′ ,𝑦
New tree: 𝑓 𝑖 −𝜕𝐿 𝑔 𝜃 𝐺,𝑋 ′ ,𝑦 𝜕𝑋′
[Figure]
Input: Graph G , node features X , targets Y
Initialize GBDT targets Y =
Y for epoch
i = 1 to N do # Train k trees of GBDT model
f i ←− k arg min f
i
L ( f i ( X ) , Y ) with eq.
f ← f + f i # Train l steps of GNN on new node features
X (cid:48)
← f ( X ) θ, X (cid:48)
←− l arg min θ, X (cid:48)
L ( g θ ( G, X (cid:48) ) , Y )
# Update targets for next iteration of GBDT
Y ← X (cid:48) − f ( X ) end for
Output: Models GBDT f and GNN g θ
Gradient boosting approach is successful for learning on tabular data; however, there are challenges of applying GBDT on graph-structured data: (i) how to propagate relational signal, in addition to node features, to otherwise inherently tabular model; and (ii) how to train it together with GNN in end-to-end fashion.
Indeed, optimizations of GBDT and GNN follow different approaches: the parameters of GNN are optimized via gradient descent, while GBDT is constructed iteratively and the decision trees remain ﬁxed after being constructed (decision trees are based on hard splits of the feature space which makes them non-differentiable).
A straightforward approach is to train the GBDT model only on the node features and then use the obtained predictions of GBDT, jointly with the original input, as new node features for GNN.
In this case, the graph-insensitive predictions of GBDT will further be reﬁned by a graph neural network.
This approach (which we call Res-GNN ) can already boost the performance of GNN for some tasks.
However, in this case, GBDT model completely ignores the graph structure and may miss descriptive features of the graph, providing inaccurate input data to GNN.
In contrast, we propose end-to-end training of GBDT and GNN called BGNN (for Boost-GNN).
As before, we ﬁrst apply GBDT and then GNN, but now we optimize both of them taking into account the quality of ﬁnal predictions.
The training of BGNN is shown in Figure 1.
Recall that already built decision trees cannot be properly tuned due to their discrete structure, so we iteratively update GBDT model by adding new trees such that they approximate GNN loss function.
In Algorithm 1 we present the training of BGNN model that combines GBDT and GNN for any node-level prediction problem such as semi-supervised node regression or classiﬁcation.
In the ﬁrst iteration, we build a GBDT model f 1 ( x ) with k decision trees by minimizing the loss function
L ( f 1 ( x ) , y ) averaged over the train nodes, following equation (2).
Using all predictions f 1 ( X ) , we update the node features to X (cid:48) that we pass to GNN.
Possible update functions that we experiment with include concatenation with the original node features and their replacement by f 1 ( X ) .
Next, we train a graph neural network g θ on a graph G with node features X (cid:48) for all nodes by minimizing L ( g θ ( G, X (cid:48) ) , Y ) with l steps of gradient descent.
Importantly, we optimize both the parameters θ of GNN and the node features X (cid:48) .
Then, we use the difference between the optimized node features X (cid:48) new and the input node features X (cid:48)
=
f 1 ( X ) as the target for the next decision trees built by GBDT.
If l = 1 , the difference X (cid:48) new − X (cid:48) exactly equals the negative gradient of the loss function w.r.t.
the input features X (cid:48) multiplied by the learning rate η :
[Equation]
In the second iteration, we train a new GBDT model f 2 , with the original input features X but new target labels: X (cid:48) new − X (cid:48) .
Intuitively, f 2 ﬁts the direction that would improve GNN prediction based on the ﬁrst predictions f 1 ( X ) .
In other words, GBDT approximates the gradient steps made by GNN for the node features X .
After f 2 is trained, we combine the predictions f
( X ) =
f 1 ( X ) + f 2 ( X ) and pass the obtained values to GNN as node features.
GNN model g θ again does l steps of backpropagation and passes the new difference X (cid:48) new − X (cid:48) as a target to the next iteration of GBDT.
In total, the model is trained for
N epochs and outputs a GBDT model f : X (cid:55)→ Y and GNN model g θ : ( G, X ) (cid:55)→ Y , which can be used for downstream tasks.
Intuitively, BGNN model consists of two consecutive blocks, GBDT and GNN, which are trained end-to-end, and therefore can be interpreted from two angles: GBDT as an embedding layer for GNN or GNN as a parametric loss function for GBDT.
In the former case, GBDT transforms the original input features X to new node features X (cid:48) which are then passed to GNN.
In the latter case, one can see BGNN as a standard gradient boosted training where GNN acts as a complex loss function that depends on the graph topology.
E XPERIMENTS
We have performed a comparative evaluation of BGNN and Res-GNN against a wide variety of strong baselines and previous approaches on heterogeneous node prediction problems, achieving signiﬁcant improvement in performance across all of them.
This section outlines our experimental setting, the results on heterogeneous node regression problems, and extracted feature representations.
In our ﬁrst experiments, we want to answer two questions:
Q1 Does combination of GBDT and GNN lead to better qualitative results in heterogeneous node regression and classiﬁcation problems?
Q2 Is end-to-end training proposed in Algorithm 1 better than a combination of pretrained GBDT with GNN?
To answer these questions, we consider several strong baselines among GBDTs, GNNs, and pure neural networks.
CatBoost is a recent GBDT implementation (Prokhorenkova et al., 2018) that uses oblivious trees as weak learners.
LightGBM is another GBDT model (Ke et al., 2017) that is used extensively in ML competitions.
Among GNNs, we tested four state-of-the-art recent models that showed superior performance in node prediction tasks: GAT (Veliˇckovi´c et al., 2018), GCN (Kipf &
Welling, 2016), AGNN (Thekumparampil et al., 2018), APPNP (Klicpera et al., 2018).
Additionally, we test the performance of fully-connected neural network FCNN and its end-to-end combination with GNNs, FCNN - GNN .
We compare these baselines against two proposed approaches: end-to-end model BGNN and not endto-end model Res-GNN .
BGNN model follows Algorithm 1 and builds each tree approximating the
GNN error in the previous iteration.
In contrast, Res-GNN ﬁrst trains a GBDT model on the training set of nodes and then appends its predictions of all nodes in a graph to the original node features, after which GNN is trained on the updated features, and GNN’s predictions are used to calculate metrics.
Hence, Res-GNN is a two-stage approach where the training of GBDT is independent of
GNN.
On the other hand, BGNN trains GBDT and GNN simultaneously in an end-to-end fashion.
We ensure that the comparison is done fairly by training each model until the convergence with a reasonable set of hyperparameters evaluated on the validation set.
We run each hyperparameter setting three times and take the average of the results.
Furthermore, we have ﬁve random splits of the data, and the ﬁnal number represents the average performance of the model for all ﬁve random seeds.
More details about hyperparameters can be found in Appendix B.
4. H ETEROGENEOUS NODE REGRESSION 4.1. D ATASETS
We utilize ﬁve real-world node regression datasets that have different properties, outlined in Table 1.
Three of these datasets, House , County , and VK , are balanced.
The distributions of the other two,
Wiki and Avazu , have a heavy tail of target variables and hence are imbalanced.
All datasets have a heterogeneous setting with the features of different scale, nature, and missing values.
Additional details about the datasets can be found in Appendix C
[Table]
[Table] [Table]
R ESULTS
The results of our comparative evaluation for node regression are summarized in Table 2.
We report the mean RMSE (with standard deviation) on the test set and the relative gap between RMSE of GAT model (Veliˇckovi´c et al., 2018) and other methods, i.e., gap = ( r m − r gnn ) /r gnn , where r m and r gnn are RMSE of that model and of GAT , respectively.
Our results successfully demonstrate signiﬁcant improvement of BGNN model across all baselines and datasets.
In particular, for the balanced case BGNN achieves 10%, 14%
[Table]
and 5% reduction of the error for House , County [Table] and VK datasets, respectively.
Res-GNN model that uses pretrained
CatBoost model for the input of GNN also decreases RMSE, although not as much as the end-to-end model BGNN .
Likewise, BGNN model excels on imbalanced datasets having 26% and 6% decrease of RMSE for Wiki and Avazu , respectively.
This shows that BGNN leads to better qualitative results
[Table]
and its end-to-end training outperforms other approaches in node prediction tasks for graphs with tabular data .
We can also conclude that the end-to-end combination FCNN - GNN leads to better performance than of using pure GNN ; however, its improvement is smaller than for BGNN model that uses the advantages of GBDT models.
Moreover, CatBoost , LightGBM , and FCNN can be effective on their own, but their performance is not stable across all datasets.
Overall, these experiments demonstrate the superiority of BGNN model against other strong models. 4. N ODE CLASSIFICATION
For node classiﬁcation, we use ﬁve datasets with different properties.
Due to the lack of publicly available datasets with heterogeneous node features, we adopt datasets House and VK from the regression task by converting target labels into several discrete classes.
We additionally include two sparse node classiﬁcation datasets SLAP and DBLP coming from heterogeneous information networks (HIN), where nodes have several different types.
For completeness, we also include one homogeneous dataset OGB-ArXiv (Hu et al., 2020a).
We note that the node features in this dataset correspond to a 128-dimensional feature vector obtained by averaging the embeddings of words in its title and abstract. Hence, the features of this dataset are not heterogeneous, and therefore GBDT is not expected to be high compared to neural network approaches. More details about this dataset can be found in Appendix D. The experimental results for node classiﬁcation are summarized in Table 3.
We report the mean accuracy (with standard deviation) on the test nodes and the relative gap between accuracy of GNN model ( GAT in this experiment) and other methods,
i [Table]
e., gap =
( a m − a gnn ) /a gnn , where a m and a gnn are accuracies of that model and of GNN respectively.
[Table]
[Table]
As can be seen from the Table 3, results on the datasets with tabular features ( House and VK ) favor our approach BGNN with a signiﬁcant margin.
For example, for VK dataset BGNN achieves almost 14% of relative increase in accuracy metric.
This demonstrates that learned representations of GBDT together with GNN can be equally useful for node classiﬁcation setting on data with heterogeneous features.
The other two datasets, Slap and DBLP , have sparse bag-of-words features that are particularly challenging for GNN model.
On these two datasets, GBDT is the strongest baseline.
Moreover, since FCNN outperforms GNN , we conclude that graph structure does not help, hence BGNN is not supposed to beat GBDT .
This is indeed the case: the ﬁnal accuracy of BGNN is slightly worse than that of GBDT .
The last dataset OGB-ArXiv has homogeneous node features (embeddings of words) that were pretrained by another neural network.
Consequently, the original GNN model achieves the top performance followed by Res-GNN and BGNN models.
In a nutshell, GBDT does not learn good predictions on the homogeneous input features and therefore reduces discriminative power of GNN .
Both cases, with sparse and with homogeneous features, show that performance of BGNN is on par or higher than of GNN ; however, lacking heterogeneous structure in the data may make the joint training of GBDT and GNN redundant.
4. C ONSISTENCY ACROSS DIFFERENT GNN MODELS
Seeing that our models perform signiﬁcantly better than strong baselines on various datasets, we want to test whether the improvement is consistent when different GNN models are used.
In particular, we ask:
Q3 Do different GNN models beneﬁt from our approach of combination with GBDT model?
To answer this question we compare GNN models that include GAT (Veli ˇ ckovi ´ c et al., 2018), GCN (Kipf & Welling, 2016), AGNN (Thekumparampil et al., 2018), and APPNP (Klicpera et al., 2018).
We substitute each of these models to Res-GNN and BGNN models and measure the relative change in performance with respect to the original GNN’s performance.
[Figure]
In Figure 2 we report the RMSE gap between Res-GNN and BGNN for each of the GNN architectures, i.e., we compute gap =
( r m − r gnn ) /r gnn , where r m and r gnn are RMSE of that model and of
GNN respectively.
This experiment positively answers Q3 showing that all tested GNN architectures signiﬁcantly beneﬁt from our proposed approach .
For example, for House dataset the decrease in the mean squared error is 10%, 15%, 15%, and 16% for GAT , GCN , AGNN , and APPNP models respectively.
Additionally, one can see that end-to-end training of BGNN (red, squared) leads to larger improvements than a na¨ıve combination of CatBoost and GNN in Res-GNN model (yellow, diagonal).
Exact metrics and running time are in Appendix E.
4. R UNNING TIME
As previous experiments demonstrated superior quality across various datasets and GNN models, it is important to understand if the additional GBDT part can become a bottleneck in terms of efﬁciency for running this model on real-world datasets.
Hence, we ask:
[Equation] BGNN and Res-GNN models incur a signiﬁcant increase in running time?
To answer this question, we measure the running time until obtaining the ﬁnal results for each model and also plot RMSE on the test set during training for all models (with winning hyperparameters).
Table 4 presents running time for each model.
We can see that both BGNN and Res-GNN run faster than GNN in most cases.
In other words, BGNN and Res-GNN models do not incur an increase in running time but actually are more efﬁcient than GNN .
For example, for VK dataset BGNN and
Res-GNN run 3x and 2x faster than GNN , respectively.
Moreover, BGNN model is consistently faster than another end-to-end implementation of FCNN - GNN model that uses FCNN instead of
CatBoost to preprocess original input features.
[Table]
The reason for improved efﬁciency is that BGNN and Res-GNN converge with a much fewer number of iterations as demonstrated in Figure 3.
We can see that BGNN converges within the ﬁrst ten iterations (for k = 20 ), leading to fast training.
In contrast, Res-GNN is similar in terms of convergence to GNN for the ﬁrst 100 epochs, but then it continues decreasing loss function unlike
GNN that requires much more epochs to converge.
This behavior is similar in other datasets (see
Appendix F).
[Figure]
4. V ISUALIZING PREDICTIONS
To investigate the performance of BGNN we plot the ﬁnal predictions of trained models for observations in the training set. Our motivation is to scrutinize which points are correctly classiﬁed by different models. Figure 4 displays the predictions of GBDT , GNN , Res-GNN , and BGNN models as well as the true target value. To better understand BGNN model, in Figure 4(e)
we show the values predicted by GBDT that was trained as a part of BGNN . This experiment is performed on House dataset, the plots for other datasets show similar trends and can be found in the supplementary materials.
Several observations can be drawn from these ﬁgures. First, the true target values change quite smoothly within local neighborhoods; however, there are a few outliers: single red points among many blue points and conversely. These points can mislead the model during the training, predicting the wrong target value for many observations in the local neighborhoods of the outliers. Hence, it is important for a model to make smoothed predictions in the local neighborhoods.
Second, [Figure] comparing the prediction spaces of GBDT , GNN , Res-GNN , and BGNN models we can observe that predictions for GBDT are much more grainy with large variations in the neighborhoods of the vertices (high quality images can be found in the supplementary materials). Intuitively, because the
GBDT model does not have access to the graph structure, it cannot propagate its predictions in the nodes’ vicinity. Alternatively, GNN , Res-GNN , and BGNN can extrapolate the outputs among local neighbors, smoothing out the ﬁnal predictions as seen in Figures 4(c), 4(d), 4(f). Third, focusing on the values of predictions (color bars on the right of each plot) of GBDT , GNN , and BGNN models we notice that the scale of ﬁnal predictions for GBDT and BGNN models is closely aligned with the true predictions, while GNN ’s predictions mismatch the true values by large margin. Our intuition is that the expressive power of GBDT to learn piecewise decision boundaries common in tabular datasets helps GBDT and BGNN to properly tune its ﬁnal predictions with respect to the true range of values, while GNN relies solely on neural layers to learn complex decision rules.
Another observation comes from looking at the values predicted by GBDT trained as a part of BGNN (see Figure 4(e)). While this GBDT model is initialized using the true target labels, during the training it was not forced to predict the target. Interestingly, this model shows the same trend and clearly captures the regions on high/low target values.
On the other hand, GBDT trained as a part of BGNN is much more conservative: on all datasets, the range of predicted values is signiﬁcantly smaller than the true one. We hypothesize that GBDT is trained to scale its predictions to make them more suitable for further improvements by GNN . Also, it is clearly seen that GBDT inside BGNN has large variations for some points (predicts high values for some examples with small targets and vice versa), but these cases are then successfully corrected by GNN . We notice the same patterns for other datasets studied in this paper.
C ONCLUSION
We have presented BGNN , a novel architecture for learning on graphs with heterogeneous tabular data.
BGNN ﬁrst takes advantages of the GBDT model to build hyperplane decision boundaries that are common for heterogeneous data, and then utilizes GNN to reﬁne predictions using relational information.
Our approach is end-to-end and can be incorporated with any message-passing neural network and gradient boosting method.
Extensive experiments demonstrate that the proposed architecture is superior to strong existing competitors in terms of accuracy of predictions and running time.
A promising research direction is to extend this approach to graph-level predictions such as graph classiﬁcation or subgraph detection.
A F URTHER RELATED WORK
To the best of our knowledge, there are no approaches combining the beneﬁts of GBDT and GNN models for representation learning on graphs with tabular data.
However, there are many attempts to adapt non-graph neural networks for tabular data or to combine them with gradient boosting in different ways.
Several works (Popov et al., 2019; Yang et al., 2018; Zhou & Feng, 2019; Feng et al., 2018; Hazimeh et al., 2020) attempt to mitigate the non-differentiable nature of decision trees.
For example, Popov et al.
(2019) proposed to replace hard choices for tree splitting features and splitting thresholds with their continuous counterparts, using α -entmax transformation (Peters et al., 2019).
While such an approach becomes suitable for a union of decision trees with GNN, the computational burden of training both end-to-end becomes a bottleneck for large graphs.
Another method (Badirli et al., 2020) uses neural networks as weak learners for the GBDT model.
For graph representation problems such as node regression, one can replace standard neural networks with graph neural networks.
However, training different GNN as weak classiﬁers at once would be exhaustive.
Additionally, such a combination lacks some advantages of GBDT, like handling heterogeneous and categorical features and missing values.
An approach called AdaGCN (Sun et al., 2019) incorporates AdaBoost ideas into the design of GNNs in order to construct deep models.
Again, this method does not exploit the advantages of GBDT methods.
Finally, Li et al.
(2019) investigated different ways of combining decision-tree-based models and neural networks.
While the motivation is similar to ours — get the beneﬁts of both types of models — the paper focuses speciﬁcally on learning-to-rank problems.
Additionally, while some of their methods are similar in spirit to Res-GNN , they do not update GBDT in an end-to-end manner, which is a substantial contribution of the current research.
B H YPERPARAMETERS
For GBDT models we have the following hyperparemters.
LightGBM has 31 leaves, 0.9 feature fraction, 0.8 bagging fraction, 5 bagging frequency, and learning 0.03.
CatBoost has depth 8 and learning rate 0.03.
Our experiments didn’t show signiﬁcant variation among different hyperparameters, except learning rate, which we selected using hyperparameter tuning.
For GNN models we use the following hyperparameters.
All neural models are initialized using
Glorot initialization (Glorot & Bengio, 2010) and trained to minimize RMSE (regression) and cross-entropy on the training nodes using the Adam SGD optimizer (Kingma & Ba, 2014).
In all cases we use an early stopping strategy on RMSE (regression) or Accuracy (classiﬁcation) on the validation nodes, with a patience of 100 epochs (except for BGNN where we use 20 epochs due to larger number of updates in one epoch).
GAT , GCN , and AGNN models have two convolutional layers with dropout and ELU activation function (Clevert et al., 2016).
APPNP has a two-layer fully-connected neural network with dropout and ELU activation followed by a convolutional layer with k = 10 and alpha = 0 .
We use eight heads with eight hidden neurons for GAT model and 64 hidden neurons for other GNN models.
For Res-GNN and BGNN models we use CatBoost ’s predictions together with GAT model.
For fully-connected neural networks FCNN and FCNN - GNN we experiment with three layers, with dropout and ReLU activation function.
We have a hidden dimension of 128.
End-to-end combination of FCNN and GNN appends the predictions of FCNN to the original input data, which is then passed to GNN model.
All models are trained until convergence with the early stopping of 300 epochs.
For neural networks, we perform a hyperparameter search on learning rate in the range [0.1, 0.01, 0.005].
Additionally, we run hyperparameters on the number of updates k and l for BGNN in the range [1,5,10,15,20] and whether to concatenate predictions of GBDT with the original input features.
Every hyperparameter setting is evaluated three times and an average is taken.
We use ﬁve random splits for train/validation/test with 0.6/0.2/0.2 ratio (for regression) and 800/400/remaining (for classiﬁcation).
The average across ﬁve seeds is reported in the tables.
C R EGRESSION DATASETS
In House dataset (Pace & Barry, 1997) nodes are the properties, edges connect the proximal nodes and the target is the price of the property.
We use publicly available dataset (Pace & Barry, 1997) of all the block groups in California collected from the 1990 Census.
We connect each block with at most ﬁve of its nearest neighbors if they lie within a ball of a certain radius, as measured by latitude and longitude.
We keep the following node features: MedInc, HouseAge, AveRooms, AveBedrms,
Population, AveOccup .
County dataset (Jia & Benson, 2020) is a county-level election map network.
Each node is a county and two nodes are connected if they share a border; we predict the unemployment rate for a county.
We use a node regression dataset with the node features, coming from 2016 year.
These features include
DEM, GOP, MedianIncome, MigraRate, BirthRate, DeathRate, BachelorRate, UnemploymentRate .
We follow the setup of the original paper and select UnemploymentRate as the target label.
We ﬁlter out all nodes in the original data if they do not have features.
VK dataset (Tsitsulin et al., 2018) is a popular social network where people are mutually connected based on friendships and the regression problem is to predict the age of a node.
We use an open-access subsample of the VK social network of the ﬁrst 1M users 1 .
Then, the dataset has been preprocessed to keep only the users who opt in to share their demographic information and preferences: country, city, has mobile, last seen platform, political, religion id, alcohol, smoking, relation, sex, university .
Wiki dataset (Rozemberczki et al., 2019) represents a page-page network on a speciﬁc topic (squirrels) with the task of predicting average monthly trafﬁc.
The features are bag-of-words for informative nouns (3148 in total) that appeared in the main text of the Wikipedia article.
The target is the average monthly trafﬁc between October 2017 and November 2018 for each article.
Avazu dataset (Song et al., 2019) represents a device-device network, with two devices being connected if they appear on the same site within the same application.
The goal in this dataset is to predict CTR for each device.
We take the ﬁrst 10M rows from the publicly available train log of user clicks 2 .
We compute click-through-rate (CTR) for each device id and ﬁlter those ids that do not have at least 10 impressions.
We connect two devices if they had impressions on the same site id from the same application id.
The node features are anonymized categories: C1, C14, C15, C16, C17, C18,
C19, C20, C21 .
D C LASSIFICATION DATASETS
For node classiﬁcation, we consider three types of node features: heterogeneous ( VK and House ), sparse ( Slap and DBLP ), and homogeneous ( OGB-ArXiv ).
[Table]
We consider performance on two heterogeneous
House and VK datasets where we transform the original target value with respect to the bin it falls to.
More speciﬁcally, for VK we consider the classes < 20 , 20 − 25 , 25 − 30 , . .
, 45 − 50 , > 50 for the age attribute.
Similarly, for House dataset we replace the target value with the bin it falls to in the range [1 ., 1 . 5 , 2 ., 2 . 5] .
Hence, there are 7 andclasses for VK and House data.
Among datasets with sparse features we consider two datasets coming from heterogeneous information networks (HIN), where nodes have a few different types.
A common way to represent HIN is through meta-paths, i.e., a collection of all possible paths between nodes of a particular type.
For example, for citation network one may specify paths of the type paper-author-paper (PAP) and of the type paper-subject-paper (PSP).
Then the original graph is approximated as several adjacency matrices for different types.
To obtain a single graph in this work we use adjacency matrices for relations GG for SLAP and APA for DBLP , which closely reﬂect the original relationships between genes for SLAP and authors for DBLP .
Each author in DBLP has a bag-of-words representation (300 words) of all the abstracts published by an author.
Each gene in SLAP has 3000 features that correspond to the extracted gene ontology terms (GO terms).
Furthermore, for every node, we compute the degrees for all types of relations and append them as additional node features.
For example, for DBLP we have two additional node features corresponding to degrees for paper nodes in APA and APCPA adjacency matrices.
A summary of statistics is outlined in Table 5.
DBLP dataset (Ren et al., 2019) is a network with three node types (authors, papers, conferences) and four target classes of the authors (database, data mining, information retrieval, and machine learning).
SLAP dataset (Xiao et al., 2019) is a multiple-hub network in bioinformatics that contains node types such as chemical compound, gene, disease, and pathway, etc.
The goal is to predict one of 15 gene types.
For datasets with homogeneous node features we consider OGB-ArXiv (Hu et al., 2020a).
The node features in this dataset correspond to 128-dimensional feature vector obtained by averaging the embeddings of words in its title and abstract.
Hence the features of this dataset are not heterogeneous and therefore performance of GBDT is not expected to be high compared to neural network approaches. Also we note that for this particular dataset we used the implementation of GAT 3 as a backbone architecture for GNN , Res-GNN , and BGNN models.
This model scored the top place on the leaderboard 4 .
E C OMPARISON OF GNN MODELS
In this section we include exact RMSE and time (s) for all tested GNN models on all datasets.
We consider several state-of-the-art GNN models that include GAT (Veli ˇ ckovi ´ c et al., 2018), GCN (Kipf & Welling, 2016), AGNN (Thekumparampil et al., 2018), and APPNP (Klicpera et al., 2018).
Table 6 demonstrates that for all considered models BGNN and Res-GNN achieve signiﬁcant increase in performance compared to vanilla GNN .
Additionally, end-to-end training of BGNN achieves typically better results than a straightforward implementation of Res-GNN .
[Table]
F L OSS CONVERGENCE
We present RMSE of test set during training for remaining datasets, County , Wiki [Table] and Avazu .
These results conﬁrm that BGNN model converges to its optimal value within the ﬁrst ten iterations (for k = 20 ), while Res-GNN converges similar to GNN for the ﬁrst 100 iterations
and then it continues to decrease its loss function faster than GNN .
[Figure]
G L EARNED REPRESENTATIONS
We also demonstrate the effectiveness of the learned representations by plotting the ﬁrst layer of pretrained GNN and BGNN models.
Both are pretrained on the train set of House dataset with the
GAT architecture (Veli ˇ ckovi ´ c et al., 2018).
For both we extract the ﬁrst hidden layer of GNN and apply the t-SNE transformation (Maaten & Hinton, 2008) with identical parameters to obtain 2D coordinates.
We color each point based on the target value.
This representation exhibits perceptible clustering in the projected latent space for BGNN model, unlike GNN ’s embeddings which are rather scattered haphazardly.
In addition to other experiments it further illustrates the discriminative power of our approach.
[Figure]