Paper: TabGraphs: new benchmark and insights for learning on graphs with tabular features (ICLR-2024)

Reviewer comment (Experiments):
It is unclear that how much model performance can be achieved from the proposed heterogeneous features. See Q1.

Author response:
We would like to point out that for different kinds of datasets different types of features (e.g., homogeneous or heterogeneous) are available. In our work, we specifically focus on heterogeneous features not because they are meant to provide better or worse performance, but because for many real-world datasets this is the only kind of features available, and thus designing models specifically for working with such features is important. For this reason, a good benchmark of datasets with such features is required.

Reviewer comment (Experiments):
Important data is missing. Strings or texts are common data types in Tabular data. TabGraph does not include strings or texts data (see Q2). Data from science domain is not included (see Q3).

Author response:
Before feeding into an ML model, strings of text are typically preprocessed using bag-of-words and other text embedding models. These algorithms produce homogeneous features. As we have discussed in the related work section, there is already a vast number of popular datasets with such text-based homogeneous features. In our paper, we specifically focus on graphs with heterogeneous features, which can often be found in real-world tasks, but for which good datasets are currently lacking. 

Note that tabular datasets that do not have any strings of text as features are quite widespread in the real world. For example, among the datasets considered in our paper, only questions-tab had strings of text that could be used as additional features. These features were used as the only feature type in the original version of the questions dataset introduced in [1]. However, we removed them in our version of the dataset, as we have found out that, in this particular case, models perform much better with heterogeneous features, which can be seen by comparing the results of GNNs in our paper and [1] on this dataset.

Indeed, it would be very useful to introduce more datasets from different sciences, but, unfortunately, we do not have access to such datasets. We will continue looking for them, and if you are aware of any quality open scientific datasets with heterogeneous features and additional graph structure, we will be glad to include them in our benchmark.

Reviewer comment (Experiments):
Insights are not completely convincing as datasets are preprocessed by rules. See Q4 and Q5.

Author response:
In fact, all of the considered models can achieve almost perfect metrics on this dataset with the full feature set (we will clarify this in the paper). This dataset is based on the data from [2]. From the description of the data, we hypothesize that the number of votes was used for constructing the target variable, and thus introduces a data leak. Because of that, we decided to remove this feature.

The thresholds were chosen so as to produce a graph that is not too sparse and not too dense. Indeed, this threshold can be viewed as a hyper-parameter of graph construction, however, in our paper, we show that the produced graphs are helpful for the task even without searching over the values of this parameter.

Reviewer comment (Novelty):
Although PLR can boost the performance of GNN, it is not novel. It is not surprising that a better node representation can boost the performance of GNN.

Author response:
Indeed PLR is not novel (and we do not claim it to be). However, it was never used in the graph ML setting before, and our work highlights that it can be an easy way to improve performance on realistic graph datasets with heterogeneous tabular features.

Reviewer comment (Experiments):
Although the author claimed that these 8 datasets in TabGraph is diverse, their selection criteria is unclear to the reader. In fact, lots of tabular benchmarks contain hundreds of datasets (e.g., AMLB). Benchmarking on only 8 tables may not be conclusive.

Author response:
For our benchmark, we have specifically selected datasets with available external information about the relationship between data samples that can be used for constructing a graph. Such datasets are much more difficult to find than standard tabular datasets. If you are aware of other realistic tabular datasets with such external information, please let us know so that we could include them in our benchmark.

Note that, in our work, we consider different kinds of relations between data samples — they are based on interactions (e.g., in the questions-tab dataset), activity similarity (e.g., in the avazu-devices dataset), physical connection (e.g., in the city-roads dataset). Importantly, we show that all these relation types can provide useful information to the models that can leverage graph-structured data.

Reviewer comment (Experiments):
What's the selection criteria of these 8 tabular datasets?

Author response:
As mentioned above, our aim was to select datasets that have meaningful prediction tasks and contain external relational information, while being quite diverse in terms of tabular feature distribution and underlying graph structure.

Note that directly comparing samples based on their features is not trivial in the case of heterogeneous tabular features (on which we focus in our paper), as these features have widely different scales, distributions, meaning and importance for the task. However, comparing their representation from some layer of a neural network is more meaningful. This is exactly what retrieval-augmented models do (in essence, they simultaneously learn sample representations and a graph based on their similarity). We discuss these models in the related work section and include TabR, a representative example of them, in our experiments. If you know some other meaningful ways to measure the similarity between objects with heterogeneous tabular features for constructing relations between them, please let us know so that we could extend our empirical study.

Reviewer comment (Experiments):
In many real-world situations, the data comes with unseen data. The paper should also address the conversion of relations to graphs and application of gnns in those settings.

Author response:
This corresponds to the inductive graph ML setup. As we have mentioned above, in our paper, we focus on the transductive setting, which is more popular in graph ML literature. However, applying GNNs to nodes unseen during training is not difficult — we just need to add edges incident to the new nodes in the graph and run the GNN on the updated graph once. Extending the benchmark to the inductive setting would be a natural next step in future research.

Reviewer comment (Presentation):
The tables of results could be of better quality. (eg, bold-face the best results)

Author response:
Thanks for this suggestion, we will highlight the best results in our tables.

Reviewer comment (Evaluation):
On the performance comparison, how does the paper differentiate or address the difference between the transductive and inductive learning methods?

Author response:
In our paper, we focus on the transductive setting, as it is the most widespread setting in graph ML literature and can be found in many real-world tasks. Indeed, GNNs have an advantage in this setting since they have access to features and incident edges of the test nodes (but not their labels, of course). However, we believe that this advantage is fair, as it is a feature of significantly different design of GNNs and standard tabular models. Also, even standard tabular models like GBDT can potentially leverage unlabeled nodes through pseudo-labeling or other semi-supervised learning techniques. We can extend our experiments with such approaches if required.

Reviewer comment (Reproducibility):
What are the hyperparameter search space for gradient boosting methods?

Author response:
The hyperparameter search space for CatBoost:

| Parameter | Distribution |
|---|---|
| ```bagging_temperature``` | $\mathrm{Uniform}[0.0,1.0]$ |
| ```depth``` | $\mathrm{UniformInt}[3,14]$ |
| ```l2_leaf_reg``` | $\mathrm{Uniform}[0.1,10.0]$ |
| ```leaf_estimation_iterations``` | $\mathrm{Uniform}[1,10]$ |
| ```learning_rate``` | $\mathrm{LogUniform}[0.001,1.0]$ |

The hyperparameter search space for XGBoost:

| Parameter | Distribution |
|---|---|
| ```colsample_bytree``` | $\mathrm{Uniform}[0.5,1.0]$ |
| ```gamma``` | $\{0.0, \mathrm{LogUniform}[0.001,100.0]\}$ |
| ```lambda``` | $\{0.0, \mathrm{LogUniform}[0.1,10.0]\}$ |
| ```learning_rate``` | $\mathrm{LogUniform}[0.001,1.0]$ |
| ```max_depth``` | $\mathrm{UniformInt}[3,14]$ |
| ```min_child_weight``` | $\mathrm{LogUniform}[0.0001,100.0]$ |
| ```subsample``` | $\mathrm{Uniform}[0.5,1.0$ |

The hyperparameter search space for LightGBM:

| Parameter | Distribution |
|---|---|
| ```feature_fraction``` | $\mathrm{Uniform}[0.5,1.0]$ |
| ```lambda_l2``` | $\{0.0, \mathrm{LogUniform}[0.1,10.0]\}$ |
| ```learning_rate``` | $\mathrm{LogUniform}[0.001,1.0]$ |
| ```num_leaves``` | $\mathrm{UniformInt}[4,768]$ |
| ```min_sum_hessian_in_leaf``` | $\mathrm{LogUniform}[0.0001,100.0]$ |
| ```bagging_fraction``` | $\mathrm{Uniform}[0.5,1.0]$

Reviewer comment (Experiments):
Even without the relation (graph-like) information, is it advisable to apply construction of graphs with features (for instance Gaussian-kernel) and then apply one of the comparing methods?

Author response:
Note that directly comparing samples based on their features is not trivial in the case of heterogeneous tabular features (on which we focus in our paper), as these features have widely different scales, distributions, meaning and importance for the task. However, comparing their representation from some layer of a neural network is more meaningful. This is exactly what retrieval-augmented models do (in essence, they simultaneously learn sample representations and a graph based on their similarity). We discuss these models in the related work section and include TabR, a representative example of them, in our experiments. If you know some other meaningful ways to measure the similarity between objects with heterogeneous tabular features for constructing relations between them, please let us know so that we could extend our empirical study.

Reviewer comment (Presentation):
It would be good to have some distinctive markings (ex, bold-face) on the table of results.

Author response:
Thanks for this suggestion, we will highlight the best results in our tables.

Reviewer comment (Experiments):
While new datasets may be useful, the paper does not provide any tangible insights. The datasets are merely described, but there is little rationale about why they have been constructed in this way and what qualitative properties they have.

Author response:
> While new datasets may be useful, the paper does not provide any tangible insights.

In Section 5, we discuss the results of our experiments in detail and highlight the main practical insights with separate bullet points (we also briefly summarize them in the introduction). In particular, we note that one should try to convert relational information (which often exists in tabular datasets) into a graph and experiment on the resulting dataset with graph machine learning methods. In that case, standard graph neural networks augmented with embeddings for numerical features tend to provide the best results, while specialized models previously proposed in the literature do not work very well. We believe that these findings are valuable and useful, especially for those who want to understand which machine learning methods actually work in practice.

> First, the datasets are merely described, but there is little rationale about why they have been constructed in this way and what qualitative properties they have.

Our datasets were constructed to cover a wide range of areas and real-world applications where graph-structured data with tabular features appears, and thus to provide a realistic and diverse benchmark. In Section 3.2, we describe in detail various properties of the introduced graph datasets, considering the distribution of node features, various structural properties of graphs, and relations between graph structure and node labels. We also explain why the prediction tasks in our datasets are meaningful and close to real-world applications. If our discussion does not cover some important aspect of the proposed benchmark, and you have any suggestions on how we can improve our analysis, please let us know so that we could include it in the revised version.

Reviewer comment (Experiments):
The experimental study reports results of some baseline models using fixed (!) hyperparameters but does not provide any additional insight beyond pure numbers. The reviewer did not learn anything from this study.

Author response:
> Second, and more importantly, the experimental study reports results of some baseline models using fixed (!) hyperparameters, but does not provide any additional insight beyond pure numbers.

This is not correct. As described in Section 5, for the considered GBDT and tabular DL methods, we have conducted an extensive hyperparameter search using Optuna for each of our datasets. Further, we have chosen a fixed set of hyperparameters in EBBS, which the authors claim to work well across different datasets, and conducted hyperparameter search in BGNN using the original experimental pipeline. As for standard GNNs, they take significantly longer to run, so we conducted preliminary experiments in which we determined a set of hyperparameters with which these models provide strong and stable results across a range of datasets, and used it in all our experiments. Note that one of our results is that GNNs typically outperform graph-agnostic models on our benchmark, thus GNNs tend to produce the best results even with less extensive hyperparameter search.

Reviewer comment (Novelty):
This is more a dataset paper than a research paper. The methodology used in dataset construction and, more importantly, the empirical study does not enhance our understanding of graph learning.

Author response:
> This is more a dataset paper than a research paper.

Please note that we have chosen “datasets and benchmarks” as the primary area of our paper. This topic is also listed in the ICLR call for papers, which shows that our paper fits the scope of the conference. We also would like to point out that empirical ML research is impossible without good datasets, thus dataset papers are of great importance for the research community.

Reviewer comment (Novelty):
The paper positions itself as graph learning with "tabular" / "heterogenous" features, which raises misleading expectations, as it is really just graphs with a fixed set of vertex-level features.

Author response:
> The paper positions itself as graph learning with "tabular" / "heterogenous" features. This raises misleading expectations, as it's really just graphs with a fixed set of vertex-level features.

In our work, we follow the terminology used by previous studies on ML for tabular data in general [1, 2] and graphs with tabular features in particular [3, 4]. In this sense, the proposed graph benchmark represents a collection of tabular datasets with known relations between samples that can be modeled as a graph. If you are aware of other uses of the terms “tabular” or “heterogeneous” in relation to data sample features, please let us know so we could add a discussion of terminology to the paper.

[1] “Revisiting deep learning models for tabular data” in NeurIPS, 2021

[2] “On embeddings for numerical features in tabular deep learning” in NeurIPS, 2022

[3] “Boost then Convolve: Gradient Boosting Meets Graph Neural Networks” in ICLR, 2021

[4] “Does your graph need a confidence boost? Convergent boosted smoothing on graphs with tabular node features” in ICLR, 2022

Reviewer comment (Experiments):
The current submission still misses a solid discussion of available benchmarks and where they fall short.

Author response:
> A prominent example is the Pokec dataset, which is commonly used in graph learning literature and has the desired features (https://snap.stanford.edu/data/soc-pokec.html). There are >100 node classification benchmarks listed on paperswithcode, there are multiple graph benchmarking papers, and there are studies using graph generators (e.g., GraphWorld).

In fact, before conducting our study, we have carefully examined numerous graph datasets that can be found in open access and filtered them due to various reasons — most of them contain a significant amount of NaN values (e.g., the mentioned Pokec dataset), have very few (if any) tabular features (e.g., OGB and many other popular datasets, as we discuss in the related work section), have issues with formulating a meaningful prediction task, represent synthetically generated data rather than real-world graphs (e.g., the mentioned GraphWorld benchmark has synthetically generated homogeneous features), etc. The remaining datasets that are suitable for our purposes we include in adapt and include to our benchmark. Since there are only a few datasets, we had to collect new graph datasets that meet our requirements: provide meaningful tasks, contain heterogeneous tabular features, and have diverse graph structures.

Reviewer comment (Experiments):
It is not clear if the submission provides state-of-the-art baselines, which would not be "easy" to beat by subsequent work.

Author response:
> Also, the method lacks recent baselines/methods (e.g., GloGCN [A] and its predecessor Link-X appear natural, but there are also many other recent methods).

Note that we have evaluated a representative set of popular GNNs (6 models). Importantly, as shown in [1], these models typically outperform many recently proposed methods, such as the mentioned GloGNN. Further, we would like to point out that GNNs achieve the best results on our benchmark. Thus, adding even more GNN models to the comparison will not change our conclusions.

[1] “A critical look at the evaluation of GNNs under heterophily: are we really making progress?” in ICLR, 2023