Repository: snap-stanford/ogb
Issue: DGL Example for ogbn-proteins

Hi OGB team,

I've adapted the PyG example for ogbn-proteins to DGL. The implementation yields a similar performance:

|                                            | PyG |  DGL |
| ------------------------ | ---- | ----- |
| Best train ROCAUC          | 0.71 | 0.72 |
| Best validation ROCAUC | 0.68 | 0.71 |
| Best test ROCAUC           | 0.65 | 0.67 |

On average the PyG example takes 510s for training per epoch and the DGL implementation takes 140s for training per epoch.

If you have not come up with a DGL example for ogbn-proteins, I'm willing to open a PR. Thanks.

Thank you for the implementation! Nice result!
Could you elaborate a bit more about your DGL implementation, e.g., did you use GPU? Did you train the model until convergence (training accuracy seems a bit low)?
Thanks!!

Thanks. I directly adapted the PyG example you provided to DGL, so it's full graph training on CPU with the same hyperparameters. See the learning curves below. I did three random runs due to the randomness of operators in both PyG and DGL.

# Loss

<img width="700" alt="Screen Shot 2020-02-20 at 00 41 02" src="https://user-images.githubusercontent.com/19576924/74869976-e7a6fd00-5393-11ea-8404-8852320f2103.png">

The PyG curves look similar and also reach a loss of about 0.26xx after 100 epochs. I guess you can push them to 0.24xx with 150 - 200 epochs or maybe doing some learning rate scheduling.

The curves of ROCAUC tend to fluctuate a lot. I guess one reason is that binary cross entropy is only a surrogate for ROCAUC and a lower binary cross entropy does not necessarily suggest a higher ROCAUC.

# Training ROCAUC

## PyG

<img width="631" alt="Screen Shot 2020-02-20 at 03 50 26" src="https://user-images.githubusercontent.com/19576924/74870183-45d3e000-5394-11ea-96be-80dd4a197c0d.png">

## DGL

<img width="1052" alt="Screen Shot 2020-02-20 at 00 45 17" src="https://user-images.githubusercontent.com/19576924/74870316-7f0c5000-5394-11ea-9335-9f71a28aec03.png">

# Validation ROCAUC

## PyG

<img width="634" alt="Screen Shot 2020-02-20 at 03 53 51" src="https://user-images.githubusercontent.com/19576924/74870407-9fd4a580-5394-11ea-865e-39debf39ffdb.png">

## DGL

<img width="1047" alt="Screen Shot 2020-02-20 at 00 48 08" src="https://user-images.githubusercontent.com/19576924/74870506-ce528080-5394-11ea-8b58-776575913c1b.png">

# Test ROCAUC

## PyG

<img width="641" alt="Screen Shot 2020-02-20 at 04 00 02" src="https://user-images.githubusercontent.com/19576924/74870905-8122de80-5395-11ea-8b74-42ab476f7c9e.png">

## DGL

<img width="1047" alt="Screen Shot 2020-02-20 at 00 52 59" src="https://user-images.githubusercontent.com/19576924/74871013-ae6f8c80-5395-11ea-9e9d-482d358fd6db.png">


Interesting! It seems that the models are underfitting the training data... I think the initial goal would be to think about how to overfit the data. Please feel free to modify the example code as it is very preliminary. Also, please try using the additional input node features that we have just added.

Please feel free to let us know if you have questions or feedback. Thanks!

Sure! How about my opening a PR for the example I have developed so far so others can take a look and making updates later if improvement found?

@weihua916 For an update, with both node and edge features, the training ROCAUC curve becomes more stable, see below:

<img width="341" alt="GIN_both_feats" src="https://user-images.githubusercontent.com/19576924/75706687-90d8e600-5cf8-11ea-817f-de3a26856a3d.png">


Very cool. Thank you Mufei. Currently, we are improving our baseline models and benchmarking them on OGB datasets. Please stay tuned for the first major release soon, where we will release our improved implementation together with new website and paper. 

Awesome! Looking forward to that.