Repository: snap-stanford/ogb
Issue: Prediction of the ogbg-molmuv

Hi, 
Just wondering why the results of the prediction using   --dataset ogbg-molmuv with main_pyg.py is so low. 
There is an example in himp-gnn/train_muv.py with the same dataset got much better accuracy. one can easily change the partition to use dataset.get_idx_split() for himp-gnn/train_muv.py but that won't reduce the accuracy. 
Any explanations?

Thanks!

Hi! Just to clarify: do you mean that `main_pyg.py` gives much lower performance than `himp-gnn/train_muv.py` with `dataset.get_idx_split()`, i.e., scaffold split? Can you share the numbers you have gotten?

Yes, you are right I was using scaffold split.
Here is what I got:
=====Epoch 100
Training...
Iteration: 100%|███████████████████████████████████████████████████████| 582/582 [00:50<00:00, 11.60it/s]
Evaluating...
Iteration: 100%|███████████████████████████████████████████████████████| 582/582 [00:13<00:00, 41.91it/s]
Iteration: 100%|█████████████████████████████████████████████████████████| 73/73 [00:02<00:00, 34.76it/s]
Iteration: 100%|█████████████████████████████████████████████████████████| 73/73 [00:02<00:00, 34.89it/s]
Train: 0.5229, Validation: 0.1265, Test: 0.1079
Finished training!
Best validation score: 0.1733
Test score: 0.1071


Thank you. This is expected since the class balance is extremely skewed and we are using Average Precision (not ROC-AUC) as a metric. Can you also show the number for the scaffold split version of `himp-gnn/train_muv.py`?

sure, I have modified the original train_muv from
https://github.com/rusty1s/himp-gnn/blob/master/train_muv.py
to use scaffold split to do the partitioning.
the results is an average of 3 runs each run took 20 epochs.
Run 3:

Epoch: 001, Loss: 0.0496, Train: 0.7086, Val: 0.7245, Test: 0.6820
Epoch: 002, Loss: 0.0145, Train: 0.7318, Val: 0.7430, Test: 0.7180
Epoch: 003, Loss: 0.0143, Train: 0.7671, Val: 0.7391, Test: 0.7180
Epoch: 004, Loss: 0.0142, Train: 0.7889, Val: 0.7597, Test: 0.7041
Epoch: 005, Loss: 0.0138, Train: 0.7768, Val: 0.7368, Test: 0.7041
Epoch: 006, Loss: 0.0137, Train: 0.7798, Val: 0.7460, Test: 0.7041
Epoch: 007, Loss: 0.0137, Train: 0.7930, Val: 0.7473, Test: 0.7041
Epoch: 008, Loss: 0.0133, Train: 0.8126, Val: 0.7364, Test: 0.7041
Epoch: 009, Loss: 0.0131, Train: 0.8234, Val: 0.7550, Test: 0.7041
Epoch: 010, Loss: 0.0131, Train: 0.8203, Val: 0.7543, Test: 0.7041
Epoch: 011, Loss: 0.0129, Train: 0.8299, Val: 0.7658, Test: 0.7295
Epoch: 012, Loss: 0.0130, Train: 0.8267, Val: 0.7477, Test: 0.7295
Epoch: 013, Loss: 0.0126, Train: 0.8451, Val: 0.7720, Test: 0.7274
Epoch: 014, Loss: 0.0126, Train: 0.8558, Val: 0.7661, Test: 0.7274
Epoch: 015, Loss: 0.0123, Train: 0.8361, Val: 0.7489, Test: 0.7274
Epoch: 016, Loss: 0.0123, Train: 0.8523, Val: 0.7680, Test: 0.7274
Epoch: 017, Loss: 0.0122, Train: 0.8652, Val: 0.7798, Test: 0.7338
Epoch: 018, Loss: 0.0124, Train: 0.8533, Val: 0.7491, Test: 0.7338
Epoch: 019, Loss: 0.0120, Train: 0.8803, Val: 0.7803, Test: 0.7613
Epoch: 020, Loss: 0.0119, Train: 0.8796, Val: 0.7578, Test: 0.7613
===========================
Final Test: 0.7505 ± 0.0126


Thank you!

As I look closer to the code, the [evaluation](https://github.com/rusty1s/himp-gnn/blob/master/train_muv.py#L74) is not using the official OGB evaluator. It's using the ROC-AUC, rather than AP. That's why the test performance is much higher.

Closing this issue, as the root cause seems to be identified.

Right.However, for other datasets, the results from the two types of evaluation are pretty similar.
For example, if we compare the dataset: ogbg-molhiv
from main_pyg.py
Epoch: 100, Train: 0.9826, Validation: 0.8235, Test: 0.7339
Finished training!
Best validation score: 0.8511
Test score: 0.7576
and from https://github.com/rusty1s/himp-gnn/blob/master/train_ogbhiv.py
with the modification of using dataset.get_idx_split()
we can get an average of 3 runs with each run of 30 epochs,
Final Test: 0.7811 ± 0.0108
The difference is much smaller.

That's because the official metric for `ogbg-molhiv` is ROC-AUC, which happens to be used by the `himp-gnn` repo. 
In general, we highly recommend using our evaluator and data split; otherwise, the results are not directly comparable.

FYI: we used AP for datasets whose class balance is extremely skewed; for molecule datasets, they are `ogbg-molmuv` and `ogbg-molpcba`.