zhiminy/EvalEng huggingface dataset:
Repository: snap-stanford/ogb
Issue: Leaderboards for (unsupervised) node embedding

Hi, 
great initiative ! 

Are you planning (or may I propose) to add a section on unsupervised node embedding ? 

This is an  topic on which a lot of efforts went but I am not sure we have robust comparisons of the plethora of methods developed.

 





Hi, thanks!!

Unsupervised node embeddings are **models** that can be applied to node classification and link prediction. Those models can be benchmarked and compared on our current node/link datasets.

Thanks for the reply. 
I certainly understand the logic of separating "models" and "tasks". 
and I have seen node2vec, Deepwalk etc... are included in various Leaderboards. 
My own concern is that following separation "models/tasks" strictly leads to comparisons between models learning on labels and those who don't and 
also the approach taken in downstream task may not be comparable. 

But I think this could be solved by adding other information on the models 

I will close the issue 

and again: thanks a lot for the great initiative.
Repository: snap-stanford/ogb
Issue: Products-GNN does not fit on a v100
Root cause: Resource Mishandling

Hi,
Thank you for this wonderful project. 

Are there any hacks to run the Products-GNN example? [https://github.com/snap-stanford/ogb/blob/master/examples/nodeproppred/products/gnn.py](https://github.com/snap-stanford/ogb/blob/master/examples/nodeproppred/products/gnn.py)

The [README](https://github.com/snap-stanford/ogb/blob/master/examples/nodeproppred/products/README.md) says that it consumes large amounts of GPU memory, but currently, I can't fit it on a v100 GPU.

Thanks!

The `ogbn-products` full-batch GNN example will consume about 33GB of GPU memory, so it will not even fit on a v100 GPU (as described in our paper). I think you can train it on a v100 though by reducing the hidden feature dimensionality. However, this example is just meant as a simple and trivial baseline, and we heavily encourage people to evaluate their methods on `ogbn-products` in combination with some sampling techniques, *e.g.*, neighborhood sampling, Cluster-GCN or GraphSAINT style sampling, or even invent some new techniques for learning on this large-scale graph.

Thanks for your quick reply!
Repository: snap-stanford/ogb
Issue: PCA matrices of "ogbn-products"?

Hi,
Thanks again for this wonderful project.
The node features in `ogbn-products` are a PCA decomposition of the original sparse bag-of-words vectors.

Do you still have, by chance, the PCA matrices that can be used to convert between the BoW-vectors and the 100-dim vectors?
Without them, this dataset is "frozen" in its current state and cannot be used in transfer learning, learning from multiple datasets at the same time, or other linguistic baselines and analyses (for example, we cannot use pre-trained word embedding because we don't have the original bag-of-words).

Thanks a lot!
Uri

Similarly, in `ogbn-arxiv`, `ogbn-papers100M` and `ogbn-mag` - the node features are the average of the word2vec embeddings.
Can we somehow recover the original words? 

For example, to examine whether the structure is really needed - 
I would like to examine the following baseline - like the MLP baseline (without using the structure) - but aggregate the words inside every node using self-attention rather than averaging.
As I understand, it is currently impossible to do that because I cannot recover the original word vectors.

Thanks!

Hi, this is a great question!

Although we have not stored the exact bag-of-words for some dataset*1, it is possible to get the original features by examining the `mapping/` directory in the downloaded data folder. There, we have included information to map each node/edge/graph into real-word objects (e.g., each node in `ogbn-products` is mapped to ASIN, and each node in `ogbn-arxiv` is mapped to a unique identifier of Microsoft Academic), so that original features (and potentially even richer features) could be obtained.

Hope this helps!

*1: For `ogbn-product`, we directly used the PCA features given by Cluster-GCN paper, so we may need to contact the original author

Thanks a lot!
Repository: snap-stanford/ogb
Issue: SSL certificate verification problem
Root cause: External Dependency Breakage

When running

```python
from ogb.graphproppred import GraphPropPredDataset
dataset = GraphPropPredDataset(name='ogbg-molesol')
```
I receive the following error:

```python
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1108)
```
I can circumvent the error by inserting the following command at the top of `graphproppred/dataset.py`, but I don't think it is ideal:
```python
ssl._create_default_https_context = ssl._create_unverified_context
```
Since this error is intermittent, it might be a problem on my side, although I have not encountered the error before when using other packages.

Interesting. I have not encountered the issue before. It would be helpful if you could specify exactly when you get the particular error.

It is difficult for me to say. I'm on a Mac, running Python 3.8.3, requests 2.23. It occurs when I run the code in PyCharm or VS Code, but not when I execute in separate terminal. If I find out any more, I'll update the issue.

I will close it for now. Please feel free to reopen it once you find the issue.

Hi @weihua916,

I am encountering the same problem when trying to download datasets from ogb,

MWE:
```python
from ogb.nodeproppred import NodePropPredDataset
dataset = NodePropPredDataset(name = 'ogbn-arxiv')
```

I am using ogb v. 1.2.5 with python 3.6.9 on Ubuntu 18.04, getting the same error
as mentioned in the issue.

Thank you in advance!

Please update to 1.2.6

Thanks! Now it works!
Repository: snap-stanford/ogb
Issue: Custom datasets
Root cause: Unimplemented Feature Gap

When developing a new model, it's usually necessary to create some small, temporary, synthetic datasets for debugging, before experimenting with real datasets. What is the recommended way of connecting up our own custom datasets to the rest of OGB? For instance, should we create our own version of GraphPropPredDataset? 

Great question. I think the quickest way is for you to create your own version, i.e., modify the dataset and split as you like, and save them (`torch.save()`) and load them (`torch.load()`). Our OGB dataset objects need to handle downloading, preprocessing, splitting etc, and it is not so trivial to quickly add a custom dataset.

FYI: We are now working hard to create a pipeline so that externally-contributed datasets can be incorporated into OGB package.

Thank you for the quick response. I modified the ogbg-molhiv example to create a synthetic dataset, and it seems to work fine with the gin-virtual model from PyG. I hope that your new pipeline will make it easier to create small, synthetic datasets for debugging.

A relatively easy solution which worked for me for several datasets so far is to just cut the nodes in ogb.io.read_graph_raw (e.g. take the first 1k) and adapt edge index, additional information, and train indices accordingly (i.e. filter out anything for nodes with index >1k).

I would also be interested in creating own datasets in ogb. Is there any update on the pipeline to integrate them @weihua916 ? That would be awesome!

You can use the [DatasetSaver](https://github.com/snap-stanford/ogb/blob/master/ogb/io/README.md) class.

Thanks, that's what I've been looking for @weihua916 ! 

I just tried to use the DatasetSaver class with a heterogeneous graph, setting the parameter `is_hetero=True`. This causes an error, saying that this is not implemented yet:

```
NotImplementedError: Heterogeneous graph dataset object has not been implemented for graph property prediction yet.
---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
<ipython-input-223-873c0be554a3> in <module>
      1 # constructor
      2 dataset_name = 'ogbg-cgkg'
----> 3 saver = DatasetSaver(dataset_name = dataset_name, is_hetero = True, version = 1)

~/git/ogb/ogb/io/save_dataset.py in __init__(self, dataset_name, is_hetero, version, root)
     38 
     39         if self.dataset_prefix == 'ogbg' and self.is_hetero:
---> 40             raise NotImplementedError('Heterogeneous graph dataset object has not been implemented for graph property prediction yet.')
     41 
     42         if osp.exists(self.dataset_dir):

NotImplementedError: Heterogeneous graph dataset object has not been implemented for graph property prediction yet.
```

Is there any way to work with heterogeneous graphs then?
Repository: snap-stanford/ogb
Issue: Problem of using undirected graph in ogbn-arxiv, ogbn-papers100m and ogbn-mag

Hi guys, 

I see the code on the leaderboard are using undirected graphs for ogbn-arxiv, ogbn-papers100m and ogbn-mag, and I have a question about it. 

The nodes in the three datasets are split by time (year of publication), however, using undirected graph may cause data leakage: we cannot predict the property of older papers using newer papers. So simply adding reverse edges for the full graph is not reasonable.

One possible way of doing this is to make papers in one year only see papers in the current year and previous years. Is it correct?

Many thanks.


Hi, this is a great point! In the strict sense, you are right; we indeed cannot use features of future papers to predict labels. In OGB, we do not impose that constraint for the sake of simplicity. At least for test nodes (corresponding to the newest papers), the labels are predicted based only on the papers from the same year and the previous years. At training time, researchers should feel free to exploit the temporal information so that their models generalize well at test time, in which models can only see past papers. Hope this answers your question.

Understand, thank you for the replay!
Repository: snap-stanford/ogb
Issue: OGB on DGL 0.5
Root cause: External Dependency Breakage

Hi,
Is it possible to read OGB to a DGL graph in DGL 0.5?


Yes, OGB will be updated to adapt to DGL 0.5 (possibly in a few weeks).

So for now we must stay on dgl 0.4. Thanks


@weihua916 Let us know if you need any help on the migration.

Thanks @jermainewang! I think we can manage it. We will update our OGB version to adapt to DGL v0.5, once the [pull request](https://github.com/snap-stanford/ogb/pull/60) is merged.

The package has been updated for DGL v0.5!
Repository: snap-stanford/ogb
Issue: Species Information in ogbn-proteins
Root cause: Documentation Deficiency

Hi OGB team,

According to the description of ogbn-proteins, the nodes are proteins from 8 species. I'm wondering why the species information is not included for node features. Thanks.

Hi, that's a great point. Thank you for pointing it out.
We updated both ogbn-proteins and ogbl-ppa so that species id is now included as input node features using one-hot encoding. All the edge connections and dataset splitting remain the same.

In order to use our updated datasets, please first delete the existing dataset directories and pull the latest source code and run setup.py locally. 

Please let us know if you have any further questions or feedback. Thanks!

> In order to use our updated datasets, please first delete the existing dataset directories and pull the latest source code and run setup.py locally.

Consider adding a version number to the dataset? Newer data could overwrite the older one to save  manual intervention.

Hi OGB team, I downloaded the latest version of dataset with node features but I didn't find it well.
<img width="707" alt="Screen Shot 1441-07-14 at 9 09 21 PM" src="https://user-images.githubusercontent.com/29864861/76244930-13f3c200-624c-11ea-960d-e4a067906295.png">

Oops! I just found the latest version is three instead of two! Thank you.
<img width="936" alt="Screen Shot 1441-07-14 at 9 51 59 PM" src="https://user-images.githubusercontent.com/29864861/76247404-8ff00900-6250-11ea-920d-456c6126e291.png">


Just a small suggestion. It will be better if the latest version of dataset could be posted as well so that people could check if the dataset they are working on is the expected version :)

Thank you for your interest. We are working on the version 1.0.0. Until then many datasets are not fixed, and we may modify things a lot. It is recommended to work on the OGB datasets after the major release. Thank you for your patience.

Thanks for your suggestion. What can we expect the major release?

It will be by the end of March. Thanks!

Got it~ many thanks!
Repository: snap-stanford/ogb
Issue: The correct split ratio of ogbn-products
Root cause: Documentation Deficiency

Hi~

When I check the ogbn-products dataset, I found the train/val/test split is 196615/39323/2213091. However, the split ratio reported in the ogb [paper](https://arxiv.org/pdf/2005.00687.pdf) is 10%/2%/88%.

I am wondering is this a typo in the paper, or is the provided Dataset API not correct?

Thank you.

Hi, thanks for pointing out. You are right, and it is typo in the paper. It should be (roughly) 8/2/90. We will update it in the next version.
Repository: snap-stanford/ogb
Issue: ogbn-mag will always ask for updating even though I have already updated it.
Root cause: Environment Incompatibility

Hello, every time I use PygNodePropPredDataset(name='ogbn-mag') to load the ogbn-mag dataset it would always ask if I wanna update to the newest version even if I have already upgraded to v2. Please check that, thanks!

I cannot reproduce that error; it works fine on my end with `ogb==v1.2.2`

To update the dataset, you need to press `y`. Then, the new version will be downloaded and processed, and the message won't show up from next time.
```python
>>> from ogb.nodeproppred import PygNodePropPredDataset
>>> PygNodePropPredDataset(name = 'ogbn-mag')
ogbn-mag has been updated.
Will you update the dataset now? (y/N)
y
Downloading https://snap.stanford.edu/ogb/data/nodeproppred/mag.zip
Downloaded 0.40 GB: 100%|█████████████████████████████████████████████████████████| 413/413 [00:10<00:00, 39.15it/s]
Extracting dataset/mag.zip
Processing...
Loading necessary files...
This might take a while.
Processing graphs...
100%|█████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 2231.01it/s]
Converting graphs into PyG objects...
100%|█████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 63.79it/s]
Saving...
Done!
PygNodePropPredDataset()
>>> PygNodePropPredDataset(name = 'ogbn-mag')
PygNodePropPredDataset()
```

I was using ogb==1.2.0. 
After I upgraded to 1.2.2, issue resolved. Thank you!
Repository: snap-stanford/ogb
Issue: Couldn't reproduce result on ogbn-mag using graph_saint
Root cause: External Dependency Breakage

Hello, I was running the ogbn-mag benchmark but couldn't reproduce the leaderboard performance of GraphSAINT (R-GCN aggr) with test acc 0.4751 ± 0.0022 using the code here 
https://github.com/snap-stanford/ogb/blob/master/examples/nodeproppred/mag/graph_saint.py

The final result I got was 
Run: 01, Epoch: 30, Loss: 1.6056, Train: 4.87%, Valid: 5.98%, Test: 6.70%

Please have a look. Thanks.

BTW I'm using PyG==1.6.1 and ogb==1.2.2

I was able to reproduce the results, with PyG==1.6.0, ogb==1.2.2. Can you double-check?

That's weird, still cannot reproduce on my side. Could you please confirm that you are using the default hyperparameters as below?
Namespace(batch_size=20000, device=0, dropout=0.5, epochs=30, hidden_channels=64, lr=0.005, num_layers=2, num_steps=30, runs=10, walk_length=2)

Actually, you are right. I was able to reproduce your result of low test accuracy with PyG==1.6.1. Everything works fine with PyG==1.6.0 though. 
Hi @rusty1s, this issue has something to do with PyG. Can you take a look at it? Thanks.

This one is on me, sorry! I identified the bug in PyG and fixed it in master. There was a bug that used the wrong edge feature/type information when creating subgraphs. You can upgrade to PyG master via:
```
pip install -q git+https://github.com/rusty1s/pytorch_geometric.git
```
All other experiments using GraphSAINT are unaffected.
Repository: snap-stanford/ogb
Issue: Sampling Negative Training Edges - ogbl-ddi/ppa

When sampling the negative training edges, is it cheating to make sure you don't sample positive validation or testing edges as negative training edges?

I've seen a model implementation from arxiv where negative training examples are also allowed to sample from pairs that actually HAVE an edge, but that edge is part of the validation or testing edge split. Seems like a bug, but after further thinking I am not sure whether changing this is cheating. Thank you!

Yes, during model training, you are not allowed to use any edge information from validation and testing.
Repository: snap-stanford/ogb
Issue: Fail to train on batch size = 1

Question about lines 24 & 25 of examples/graphproppred/mol/main_pyg.py:

```
        if batch.x.shape[0] == 1 or batch.batch[-1] == 0:
            pass

```
If the batch size is set to 1, then the second condition will pass on every batch. This seems like a bug. What's the purpose of this line?

This is needed to avoid a rare corner case, where a batched graph object does not contain any edges. I remember this happens in one of the molecule regression datasets. Feel free to remove it if you would like to train on batch size of 1.
Repository: snap-stanford/ogb
Issue: Expected object of device type cuda but got device type cpu for argument #3 'index' in call to _th_index_select
Root cause: Validation Gap

Hi, I'm new to PyTorch Geometric and used a function provided by ogb:

```
class Net(torch.nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.emb = AtomEncoder(16)
        self.conv1 = GCNConv(16, 16)
        self.conv2 = GCNConv(16, dataset.num_classes)

    def forward(self, x, edge_index, batch):
        x, edge_index = x, data.edge_index
        x = self.emb(x)
        x = self.conv1(x, edge_index)
        x = F.relu(x)
        x = F.dropout(x, training=self.training)
        x = self.conv2(x, edge_index)
        
        return global_max_pool(F.log_softmax(x, dim=1), batch = batch_size)
```
The line "x = self.emb(x)" gave a error below, can someone help me?

> Expected object of device type cuda but got device type cpu for argument #3 'index' in call to _th_index_select


Looks like either your data input or your model does not sit on the same device. This should be fixable by running:
```
model = model.to(device)
data = data.to(device)
```

**Edit:** `x, edge_index = x, data.edge_index` may cause that, since your using the `edge_index` of `data`, while using the `x` that is passed to the function.
Repository: snap-stanford/ogb
Issue: pre_transform not working in nodeproppred/dataset_pyg.py
Root cause: Validation Gap

https://github.com/snap-stanford/ogb/blob/b1287317029e5a1c982a1cf13b2bbb7bc61e129c/ogb/nodeproppred/dataset_pyg.py#L192

Perhaps it should be `data = data if self.pre_transform is None else self.pre_transform(data) ` ?

Thanks, you are right. Just fixed it.
Repository: snap-stanford/ogb
Issue: How to deal with large graphs
Root cause: Documentation Deficiency

Hey folks,

I'm interested in running some experiments on a larger graph like ```obgn-papers100M``` but unfortunately AWS keeps killing my process when I hook it up to https://github.com/danielegrattarola/spektral/blob/master/examples/node_prediction/ogbn-proteins_gcn.py and just change the dataset name

So before I build out my own homegrown data loader solution I was wondering how do you usually deal with this large kind of dataset

- Just get a machine with tons of RAM? About 100GB?
- Create a TF dataset or Torch dataset?
- Anything else?

Would really appreciate any pointers in the right direction


I'd suggest to use a large RAM (I do not expect one to work on `ogbn-papers100M` with an ordinary machine). Also, please make sure to use `ogb>=1.2.3`, as it is more memory efficient in processing the large dataset (see https://github.com/snap-stanford/ogb/issues/46). Hope this helps!

I'm curious how do you do training on this kind of dataset on a GPU? Since you can't really fit this into memory at all. Even with model or data parallelism I don't see a trivial solution for loading parts of the graph on multiple GPUs.

With multiple small graphs I can load multiple minibatches but with one gigantic graph I'm not sure what to do.

I'm guessing you just use a beefy CPU with tons of RAM?

And if not any recommendations for repos I can check out?

@danielegrattarola in case you're interested, I'd like to create a representative example for others to use as well for giant graphs with Spektral

I think it is indeed very difficult to train on this kind of dataset with a GPU :) There's a reason none of the current entries on the leaderboard use GPU.

Theoretically, there's no reason you can't do graph sampling methods like cluster-GCN or GraphSAINT - however, those generally increase computational time. In addition, at inference time, you probably still need to go back to CPU.

This is super helpful, thank you!

@msaroufim You can see an example on OGB-products (which is already quite big and difficult to fit onto GPU): https://github.com/snap-stanford/ogb/blob/master/examples/nodeproppred/products/cluster_gcn.py

@msaroufim working on large graphs is definitely an interesting challenge. Shoot me an email if you need help with Spektral, ideally it should be easy to translate any Torch+PyG example into TensorFlow+Spektral.

Cheers

I use 384GB (higher than 360GB) memory , do the ogbn-papers100M dataset  to dgl distribute data  what they need

and cost time about 4h
Repository: snap-stanford/ogb
Issue: Name for the method is restricted to 21 characters (instead of 30) on the Leaderboard Submission form
Root cause: Validation Gap

![image](https://user-images.githubusercontent.com/6355099/94983120-bdaedf00-050d-11eb-94e2-fdc1207bd02e.png)

Kind of a stupid issue, but I noticed that the text says the limit is 30, but the actual limit is 21. Is this intentional?

You are right, this has been fixed. Thanks!

Can I modify my submission to say `Label Propagation` then :)

Sure, I have modified your submission. I will update the leaderboard once I confirmed that enough instruction to reproduce the result is provided in README.

This Github repo is an excellent example: https://github.com/PaddlePaddle/PGL/tree/main/ogb_examples/nodeproppred/unimp

My bad - I've been updating the repo so it's been a bit in flux. 

It should be more clear now.


Great, updated to the leaderboard.

btw, I think I didn't include the node2vec parameters in my most recent submission - should I submit a new one or can I provide a correction here?



You can provide the correction here.

The number of parameters is:

`209194 + 21675904 = 21885098`

Sounds good. I cannot update the result to the leaderboard since I cannot find your `ogbn-products` code in https://github.com/Chillee/ogb_baselines. Below is your submission record. I suspect the Github link is incorrect?

10/3/2020 2:17:37 | 1.2.3 | Label Propagation | ogbn-products | 0.7278,0.0013 | 0.7414,0.0008 | Horace He (Cornell) | hh498@cornell.edu | https://github.com/Chillee/ogb_baselines | Unofficial | https://arxiv.org/abs/1609.02907 | 21885098 | GeForce RTX 2080 (11GB GPU)




ah sorry, that's meant to be `ogbn-arxiv`. Apologies for the incorrect submission - after I clicked "update" I suspected I had filled out some parts incorrectly, but couldn't check it. That would be nice :)

It should also say `GCN + residuals + node2vec`.

EDIT: Actually, I think I just started filling out one submission and switched to another submission in the middle... Sorry again!

No worries, updated to the leaderboard.

This was the motivating reason for my previous suggestion about modifying embedding features :)

Btw, for my recent paper, I computed some parameter counts for a submission that currently doesn't have them.

```
GraphSAGE on ogbn-products: 207454
```