Repository: snap-stanford/ogb
Issue: Performances of MLP and standard GNNs on graphproppred
Root cause: Documentation Deficiency

In benchmarking a few different approaches on graphproppred datasets, I got some surprising results which I thought might be interesting to others.

I'm not sure they best fit here (if not, please tell me), but it's probably the first spot people will look when first training models on OGB.

Specifically,

1. Naively incorporating bond information does not appear to increase performance on all but one task (BACE).
2. A simple MLP is surprisingly competitive to GNN approaches on most datasets, even outperforming them on one (BBBPTask)


Performance details can be found in the following table (showing ROC-AUC as computed by the OGB evaluator). Note that PCBATask is still being evaluated, having finished 80/100 runs for the GCNBlock and all runs for the MetaLayerBlock. PCBA runs are extremely slow at the moment, at ~3h/run.

|                | BACETask | BBBPTask | CLINTOXTask | HIVTask  | MUVTask  | PCBATask | TOX21Task | TOXCASTTask |
|----------------|----------|----------|-------------|----------|----------|----------|-----------|-------------|
| MetaLayerBlock | 0.748352 | 0.94185  | 0.99287     | 0.808189 | 0.795011 | 0.850489 | 0.781248  | 0.666357    |
| GCNBlock       | 0.691575 | 0.918152 | 0.99463     | 0.789897 | 0.803885 | 0.848058 | 0.782434  | 0.681095    |
| GINBlock       | 0.683883 | 0.930698 | 0.99754     | 0.822764 | 0.801272 | NaN      | 0.775268  | 0.661059    |
| GIN0Block      | 0.709341 | 0.923031 | 0.995917    | 0.790234 | 0.806313 | NaN      | 0.778939  | 0.655804    |
| GraphSAGEBlock | 0.708791 | 0.933984 | 0.99498     | 0.793706 | 0.810183 | NaN      | 0.782426  | 0.672336    |
| MLPBlock       | 0.690659 | 0.953998 | 0.989598    | 0.76105  | 0.769725 | NaN      | 0.770448  | 0.659917    |

And here's the ranking for each of the finished tasks, plus the total ranking. On average, all GNN methods perform similar (mean ranks between 2.42 and 3.57).

| Ranking        | BACETask | BBBPTask | CLINTOXTask | HIVTask | MUVTask | TOX21Task | TOXCASTTask | total    |
|----------------|----------|----------|-------------|---------|---------|-----------|-------------|----------|
| MetaLayerBlock | 1        | 2        | 5           | 2       | 5       | 3         | 3           | 3        |
| GCNBlock       | 4        | 6        | 4           | 5       | 3       | 1         | 1           | 3.428571 |
| GINBlock       | 6        | 4        | 1           | 1       | 4       | 5         | 4           | 3.571429 |
| GIN0Block      | 2        | 5        | 2           | 4       | 2       | 4         | 6           | 3.571429 |
| GraphSAGEBlock | 3        | 3        | 3           | 3       | 1       | 2         | 2           | 2.428571 |
| MLPBlock       | 5        | 1        | 6           | 6       | 6       | 6         | 5           | 5        |


That's not an exhaustive evaluation! Some details on my current setup:
- I'm doing a random search over hyperparameters, but only for 100 parameter configs
- I'm training for a fixed 100 epochs, with bs 128 and Adam (lr 0.01); no weight decay or learning decay rate (both on my ToDo list, also early stopping)
- Constructed models are all similar: A 100-dimensional bond and atom encoder (from OGB), followed by 1-5 layers with 16-128 units. These can use batchnorm, dropout, and jumping knowledge depending on the hyperparameters. After that, node features are globally mean-aggregated and a two-layer MLP outputs the final prediction.
- The GIN models use two MLP layers with 16 or 32 units; on 8/14 task x (GIN, GIN0) the best model uses 16 units.
- The MetaLayer is based on the Graph Network paper (and, of course, pt-geometric's implementation of it). Its edge model is a two-layer MLP based on edge features and adjacent node features, while the node model uses two two-layer MLPs to map node and edge feature for each adjacent node, then projecting them onto the current node (using mean aggreagation), then applying the second MLP. It can use residual connections (with appropriate up/downscaling); that's also a hyperparameter.
- The GIN result on TOX21 (0.775) fits pretty well with the graph prediction baseline @rusty1s added (0.761); differences in parameters and training suffice to explain that (128 vs 32 batch_size, his use of edge information, different learning rate)


Other interesting findings:
- About half of the best-performing GNN/task combinations use batchnorm, almost exclusively on HIV, PCBA (those that have been evaluated), and TOX21/TOXCAST.
- Only one best-performing model used dropout (MUV/MetaLayer, which doesn't perform well compared to other GNNs)
- Depth varies wildly for the best-performing HP configs: With 13 in total, 4 layers is the most common, closely folled by 1 layer (11x); the others occur between 5 and 8 times.
- 5/8 of the best MetaLayer parameter configs use the residual connection. Interestingly, those that don't also have 3, 4, and 5 layers, so this doesn't appear to be a necessity for deeper layers.
- On BACE, BBBP, MUV, and both TOX21 and TOXCast, overfitting seems to be very easy. Regularization might dominate here.

I'm wondering whether you had similar experiences when training the benchmarking models.

Hi Frederik, Thanks for your experiments. I think these findings are definitely interesting, and something that we have not explored much. It will benefit our community to release your implementation. Also, our baseline implementation is available [here](https://github.com/snap-stanford/ogb/tree/master/examples/graphproppred/mol). Our paper will be released soon based on these implementations.

FYI: We have recently released OGB ver1.0.0. Just want to make sure you are using the newest version. We did not modify the ogbg-mol datasets much, but we did change downloading and data processing protocols.

> It will benefit our community to release your implementation. 

I plan to do so, hopefully within the next two weeks.

> FYI: We have recently released OGB ver1.0.0. Just want to make sure you are using the newest version. We did not modify the ogbg-mol datasets much, but we did change downloading and data processing protocols.

I had missed that; I'm rerunning the experiments now on pip's v1.0.0. I was previously using f51a317, which identified itself as v0.2.0.

Another interesting sidenote: I do get speedups of about 10-20% when increasing the worker number to 4 (from the non-parallel 0 workers usually used for GNNs). Another thing to investigate.

@fdiehl commenting to keep track of when/if you release code. Looking forward to it.

I've finally cleaned it up and pushed it to github, see https://github.com/fgerzer/gnn_benchmark

As usual, I'll stress that it's research code and unfinished, but I still hope it will help people.