zhiminy/EvalEng huggingface dataset:
Repository: snap-stanford/ogb
Issue: AttributeError: 'GlobalStorage' object has no attribute 'num_classes'
Root cause: Validation Gap

data.num_classes 
Traceback (most recent call last):    File ~/anaconda3/lib/python3.8/site-packages/torch_geometric/data/storage.py:50 in __getattr__     return self[key]    File ~/anaconda3/lib/python3.8/site-packages/torch_geometric/data/storage.py:70 in __getitem__     return self._mapping[key]  KeyError: 'num_classes'   During handling of the above exception, another exception occurred:  Traceback (most recent call last):    Input In [40] in <cell line: 1>     data.num_classes    File ~/anaconda3/lib/python3.8/site-packages/torch_geometric/data/data.py:362 in __getattr__     return getattr(self._store, key)    File ~/anaconda3/lib/python3.8/site-packages/torch_geometric/data/storage.py:52 in __getattr__     raise AttributeError(  AttributeError: 'GlobalStorage' object has no attribute 'num_classes'

Can you give the code to reproduce it?
Repository: snap-stanford/ogb
Issue: num_layer not passed to GNN
Root cause: Configuration Error

Hi team,

Thanks for the nice work!

I found the `num_layer` argument in graph property prediction scripts is not passed to GNN. See [line](https://github.com/snap-stanford/ogb/blob/master/examples/graphproppred/mol/main_pyg.py#L72) and [line](https://github.com/snap-stanford/ogb/blob/master/examples/graphproppred/mol/main_pyg.py#L114). This occurs at all three example tasks: code, ppa and mol, which makes tuning num_layers in command line not working as expected.

Hi, that's a great point. Thank you for pointing it out. Now, I have passed the `num_layer` argument to GNN objects in all the datasets.
Repository: snap-stanford/ogb
Issue: Small bug in save_dataset.py
Root cause: Validation Gap

When exporting my dataset, I discovered a small bug in `save_dataset.py`

https://github.com/snap-stanford/ogb/blob/358719af702957eaf14c235c972e092c2e302ebd/ogb/io/save_dataset.py#L464

Proposed change:

`meta_dict['has_edge_attr'] = str(self.has_edge_attr)`



Merged! 

Use this feature for now via
```
pip install git+https://github.com/snap-stanford/ogb.git
```
Repository: snap-stanford/ogb
Issue: dist_l2 forward error when running train_shallow_wikikgv2.sh for WikiKG90M-v2
Root cause: Environment Incompatibility

Hi, 

Thanks for your exciting contributions to the open-source KG framework!

I have followed the steps in `README.md` and `README_wikikgv2.md` to install the package and download the LSC WikiKG90M-v2 data. When I am trying out the baseline models, it reports the following error:

![Screen Shot 2022-07-02 at 5 17 34 PM](https://user-images.githubusercontent.com/13828871/177017766-66f686ea-58bd-411b-80a1-26c4ff0d388b.png)

I tried to identify the source of error, and it seems from `dist_forward()` function defined in `extlib_cuda.cpp`. 
I am using 4 TITAN RTX GPUs (24 GB each) with Driver Version: 440.100 and CUDA Version: 10.2. 

How to fix this error? Thanks!
Repository: snap-stanford/ogb
Issue: convert graph in mol dataset back to rdkit mol object
Root cause: Unimplemented Feature Gap

Is there any way to convert a graph in one of the mol datasets, for example molhiv, back to an rdkit molecule?
I'd like to visualize it, which is possible with networkx, but rdkit's visualization has much more information.

See https://github.com/snap-stanford/ogb/issues/90
Repository: snap-stanford/ogb
Issue: version check makes "from ogb.xxx import xxx" get stuck
Root cause: External Dependency Breakage

@weihua916 @rusty1s 
The version check in [version.py](https://github.com/snap-stanford/ogb/blob/master/ogb/version.py) makes `from ogb.xxx import xxx` get stuck.

With those code:
```
❯ ipython
Python 3.9.12 (main, Apr  5 2022, 06:56:58)
Type 'copyright', 'credits' or 'license' for more information
IPython 8.2.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from ogb.graphproppred import GraphPropPredDataset

```
The program is just waiting.

Without those code:
```
❯ ipython
Python 3.9.12 (main, Apr  5 2022, 06:56:58)
Type 'copyright', 'credits' or 'license' for more information
IPython 8.2.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from ogb.graphproppred import GraphPropPredDataset

In [2]: 
```

A quick workaround is to uninstall `outdated` after `ogb` is installed.

Thank you. It really helps a lot.
Repository: snap-stanford/ogb
Issue: Stuck at Validation sanity check
Root cause: Environment Incompatibility

Hi, dear ogb team. I want to take part in OGB-LSC, but now I am stuck on some problem, and I can't find any useful issues in this repo.
 
I am run example/lsc/mag240m/gnn.py and I get the following results:
Namespace(hidden_channels=1024, batch_size=1024, dropout=0.5, epochs=100, model='gat', sizes=[25, 15], in_memory=False, device='0', evaluate=False)
Global seed set to 42
#Params 4890777
GPU available: True, used: True
TPU available: None, using: 0 TPU cores
Converting adjacency matrix... Done! [627.29s]
Reading dataset... Done! [130.84s]

  | Name      | Type       | Params
-----------------------------------------
0 | convs     | ModuleList | 1.8 M 
1 | norms     | ModuleList | 4.1 K 
2 | skips     | ModuleList | 1.8 M 
3 | mlp       | Sequential | 1.2 M 
4 | train_acc | Accuracy   | 0     
5 | val_acc   | Accuracy   | 0     
6 | test_acc  | Accuracy   | 0     
-----------------------------------------
4.9 M     Trainable params
0         Non-trainable params
4.9 M     Total params
19.563    Total estimated model params size (MB)
Validation sanity check:   0%|              | 0/2 [00:00<?, ?it/s]

is this normal? thanks for your replies.

I find this issue caused by pytorch_lightning, I set num_sanity_val_steps=0
```python
trainer = Trainer(gpus=args.device, max_epochs=args.epochs,
                 callbacks=[checkpoint_callback],
                 default_root_dir=f'logs/{args.model}',num_sanity_val_steps=0)
```
but I still stuck at Trainer

Namespace(hidden_channels=1024, batch_size=1024, dropout=0.5, epochs=100, model='gat', sizes=[25, 15], in_memory=False, device='0', evaluate=False)
Global seed set to 42
#Params 4890777
GPU available: True, used: True
TPU available: None, using: 0 TPU cores
Reading dataset... Done! [147.73s]


   | Name      | Type       | Params
-----------------------------------------
0 | convs     | ModuleList | 1.8 M 
1 | norms     | ModuleList | 4.1 K 
2 | skips     | ModuleList | 1.8 M 
3 | mlp       | Sequential | 1.2 M 
4 | train_acc | Accuracy   | 0     
5 | val_acc   | Accuracy   | 0     
6 | test_acc  | Accuracy   | 0     
-----------------------------------------

4.9 M     Trainable params
0         Non-trainable params
4.9 M     Total params
19.563    Total estimated model params size (MB)
Epoch 0:   0%|                                                                                    | 0/1223 [00:00<?, ?it/s]

I find that the stuck is due to disk speed, and i will close the comment.
Thanks, ogb team.
Repository: snap-stanford/ogb
Issue: A Question about GCN Implementation in the examples

Hello, your work is such a great contribution to the community and I really appreciate it since it is of much benefit to my study. 

But for me there is a tiny question: I am confused about your implementation of GCN, why would you set a root_emb as a learnable embedding or parameter within a GCN Conv? I saw it is used to add to a linearly projected x before a relu followed by degree normalization. Or, I want to know what's the function and motivation of the variable root_emb. Is it the first time to use this variable in literature or you just follow someone's work?

I thought it was related to virtual node method, but virtual node was already implemented in GNN_node_Virtualnode, so I still don't know what's the meaning of root_emb here.

![image](https://user-images.githubusercontent.com/39686155/179404390-ad273363-af99-4b24-836a-b7a79a530d1f.png)


Hi! Glad to hear you find OGB useful!

Note that the original GCN cannot incorporate edge feature information. Therefore, we extend the original GCN by making the message `norm.view(-1, 1) * F.relu(x_j + edge_attr)` (see [here](https://github.com/snap-stanford/ogb/blob/master/examples/graphproppred/mol/conv.py#L61)) instead of `norm.view(-1, 1) * x_j`.
Then, you can think of `self.root_emb` as `edge_attr` analogue of the self-edge.


@weihua916 Thanks for your such an prompt reply. Now I see maybe you want an analogue of `x_j + edge_attr` for self-loop (since self-loop is a default setting for a GCNConv), so you add `x_i+edge_attr` extra after aggregation of many `x_j + edge_attr` to realize it, where `edge_attr` for `x_i`'s self-loop is not in raw data.edge_attr, so you just make it be a learnable or trainable. 
It helps a lot. Thanks, again =v=

Exactly, that's the intention. You are welcome.
Repository: snap-stanford/ogb
Issue: Unable to import PygGraphPropPredDataset
Root cause: Environment Incompatibility

Dear Team Members,

Thank you very much for all the efforts. However, I am having some issues whenever I try to **import PygGraphPropPredDataset**. I have attached a screenshot for your consideration. Could you please help me out?

**Configuration:** 

- Python 3.7.3
- PyTorch 1.4.0
- DGL 0.4.3 
- Numpy 1.18.4
- pandas 1.0.1
- urllib3 1.25.9
- scikit-learn 0.22.1

**Screenshot:**

![Capture](https://user-images.githubusercontent.com/6106868/84592228-d19eda00-ae65-11ea-860d-3174abeca8f6.PNG)

I look forward to hearing from you soon.

Thank you,
@ashikrafi 

That looks like a faulty PyG installation. How did you install PyG/`torch-sparse`? Are you using the wheels or did you install via `torch-sparse`? In the latter case, please share installation logs with me. We can discuss this further at the PyG issue page, *e.g.* [here](https://github.com/rusty1s/pytorch_geometric/issues/1233). 

Hi @rusty1s 

Please have a look into it.
- https://github.com/rusty1s/pytorch_geometric/issues/1233#issuecomment-643756517

Thanks
Repository: snap-stanford/ogb
Issue: Feedback when submit results
Root cause: Documentation Deficiency

Hi,

I am preparing for the large-scale challenges on KG, and I wander if I can get any feedback when I submit results on test-dev to the leaderboard of WikiKG90Mv2? Furthermore, can I get any feedback if the results on test-challenge is submitted?

Thanks very much!

Hi! The test-dev result will be available within a week after your submission. We cannot give results on the test-challenge until the final competition evaluation.

Thank you for your reply,

one more question, I find that only the person who fills in the google form will receive the receipt email, is it the normal status? Do other teammates need any confirmations?

Only the person who fills in the form will receive the receipt email.
Repository: snap-stanford/ogb
Issue: Error when loading ogbn-mag dataset with DglNodePropPredDataset
Root cause: Validation Gap

Hi and thanks for your work,
when i use DGL Loader for ogbn-mag I receive this error: 
raise DGLError('Edge type "{}" does not exist.'.format(etype))
dgl._ffi.base.DGLError: Edge type "paper" does not exist.

![image](https://user-images.githubusercontent.com/42349814/85257650-6ebbcd00-b47b-11ea-97df-9d523e58628a.png)


Hi, thanks for noticing the bug, which is just fixed in the master branch. We will update the package this weekend together with other fixes. If you want to give it a try earlier, please install the master branch.

Thanks again for the issue, and my apologies for the mistake.

Thank you for fixing.


The package has been updated to v1.2.1. The new package version should allow you to work with `ogbn-mag` in DGL.
Repository: snap-stanford/ogb
Issue: cannot import name 'PCQM4Mv2Dataset' from 'ogb.lsc'
Root cause: Environment Incompatibility

from ogb.lsc import PCQM4Mv2Dataset

I have installed torch 1.11, rdkit 2022.03.5 and ogb 1.3.4

Interesting. I'd suggest you to replace [these lines](https://github.com/snap-stanford/ogb/blob/master/ogb/lsc/__init__.py#L16-L19) with 
```python
from .pcqm4mv2 import PCQM4Mv2Dataset, PCQM4Mv2Evaluator
```
and see the detailed error. Most likely, some library is missing or not properly installed.

I already got it, really dont know why i wasnt able to use this one, maybe it was what you've said. Thanks btw.
Repository: snap-stanford/ogb
Issue: Unable to use the new dataset ddi
Root cause: Environment Incompatibility

Hi,

I tried to test the performance of GCN on the new link prediction dataset ddi
and got the following error

![Capture](https://user-images.githubusercontent.com/20626384/85289122-97ee5480-b49f-11ea-946c-d6e598bed083.PNG)


Thank you,
Omri

I think the issue is due to the use of the old package version. Could you update the package to 1.2.0 and try again?
Repository: snap-stanford/ogb
Issue: All Results are 0.0 for ogbg-code2 example
Root cause: External Dependency Breakage

I tried to reproduce result of ogbg-code2, but at every epoch, I got 0.0 for precision, recall and F1 score and for train, validation and test, as the following image shows.
![image](https://user-images.githubusercontent.com/39686155/188317749-265649d9-feb9-43ba-9735-cd3fdff98a16.png)

The issue occurred when running in env where torch==1.10.0+cu113, torch-geometric==2.0.4 and ogb==1.3.3

I try running in env where torch==1.8.0+cu111 and torch-geometric==2.0.4 
training and evaluating seemed correctly

=====Epoch 1
Training...
Iteration: 100%|█████████████████████████████████████████████████████| 3188/3188 [06:03<00:00,  8.78it/s]
Average training loss: 3.1347407830255096
Evaluating...
Iteration: 100%|█████████████████████████████████████████████████████| 3188/3188 [02:30<00:00, 21.16it/s]
Iteration: 100%|███████████████████████████████████████████████████████| 179/179 [00:09<00:00, 17.99it/s]
Iteration: 100%|███████████████████████████████████████████████████████| 172/172 [00:09<00:00, 18.35it/s]
{'Train': {'precision': 0.11241788732670548, 'recall': 0.05771350526060361, 'F1': 0.07280211499814168}, 'Validation': {'precision': 0.09887364684226672, 'recall': 0.04849786534824006, 'F1': 0.06207377717216873}, 'Test': {'precision': 0.0993560537026912, 'recall': 0.05024431846124028, 'F1': 0.06371399366888705}}


I just want to know why?

ps: torch==1.10.2+cu113 and torch==1.11.0+cu113 both is ok, too

Then maybe it is only 1.10.0 that incurred this bug?

That's interesting. Maybe you can see what's fixed from 1.10.0 to 1.10.2

Closing this issue for now, just make sure people use 1.10.2
Repository: snap-stanford/ogb
Issue: Best model selection based on a subset of validation edges for OGBL datasets
Root cause: Documentation Deficiency

Hi OGB team,

I'm developing new submissions on OGBL datasets. Many existing submissions evaluate the model on the full validation and test sets each epoch and choose the best model parameters corresponding to the best validation metric. As the model I'm working on is very costly, can I choose the best model parameters by evaluation on a subset of the validation edges and using a different K for Hits@K? I will still report the required metrics on the full validation and test sets with the best model selected.

Thanks,
Yakun

Yes, that's no problem!
Repository: snap-stanford/ogb
Issue: [Feature Request] Using parquet files instead/alongside torch splits
Root cause: Unimplemented Feature Gap

Hello devs.
I am trying to develop support for OGB Datasets in [MLDatasets.jl](https://github.com/JuliaML/MLDatasets.jl). One of the bottlenecks we are facing is loading the .pt files. This implementation [here](https://github.com/JuliaML/MLDatasets.jl/pull/172) using Pickle.jl hack results in substantial memory usage compared to python. With new support for TorchArrow can you support parquet files for loading the splits?

Hi! Are the split files so large? They are just storing the split indices, no?

I was asking if it was possible/planned to use a language independent format to store the computed splits.

I see. That'd require all zipped files to be re-created. I do not think we will support this in the immediate future. You can probably consider some workaround on your side.

Makes sense!🙁
Repository: snap-stanford/ogb
Issue: GIN examples does not seem to work anymore
Root cause: External Dependency Breakage

I tried out running `python main_pyg.py` using Python `3.10.4` Pytorch Geometric `2.0.4` from https://github.com/snap-stanford/ogb/tree/master/examples/graphproppred/mol. However, I do not get any output. 


Solved by installing from master.



Glad to know that. Closing it for now. Please share here if you know what caused the issue.
Repository: snap-stanford/ogb
Issue: Potential exploding gradient issue
Root cause: Validation Gap

Hi OGB team,

Thanks for the great work in creating the benchmark! 

While I am trying ogbl-ppa for my own research work, I find that the existing full batch training for GCN will frequently lead to sudden performance drop with a largely increased loss, which I suspect is due to the exploding gradient. I manage to alleviate the issue by gradient clipping, i.e.,
clid_grad_norm_(model.parameters(), 1.0)
clid_grad_norm_(predictor.parameters(), 1.0)
The performance of GCN improves from ~11% (the same as in the paper) to ~18%. 

I haven't got to check whether other datasets have the same problem, but I think it is better to be aware of the issue : )

 

Thanks for the hint! We also noticed sudden performance drops, but didn't check whether this can be prevented by the usage of gradient clipping. I will run a few tests and update the examples accordingly.

I added gradient clipping to the `ppa`, `ddi` and `collab` examples. We will update our results this weekend. Once again, thanks for this great suggestion :)
Repository: snap-stanford/ogb
Issue: Evaluate rmse might be wrong?
Root cause: Algorithmic Error

https://github.com/snap-stanford/ogb/blob/c8efe8ec99d11279c80f2bcdbe1567675c1c5666/ogb/graphproppred/evaluate.py#L218

`rmse_list.append(np.sqrt(((y_true[is_labeled, i] - y_pred[is_labeled, i])**2).mean()))` should be correct.

Thanks for pointing out! You are right. This is fixed at https://github.com/snap-stanford/ogb/commit/dd9e3d0ed642619528e90d312bc8b52476338c64

Btw, this should not affect any existing results, because luckily all our regression tasks are single-task (not multi-task).
Repository: snap-stanford/ogb
Issue: A problem of loading large graph
Root cause: Unimplemented Feature Gap

Hi! When I load the large graph such as `ogbn-papers100M`, I find the size of graph is too large to load. So I'm wondering whether ogb provide functions, with which we just load the structure of graph (edge list of graph), but don't load the feature vectors of each nodes on the graph into memory. As we know, the feature size will occupy a large proportion of the size of graph.

Hi! Really good point. Unfortunately, we do not provide such functionality for the OGB datasets, but we do for the OGB-LSC datasets.

OK, thank you very much
Repository: snap-stanford/ogb
Issue: [RFC] Add system metrics to the leaderboard
Root cause: Unimplemented Feature Gap

Hi OGB folks! With more and more GNN models being proposed and submitted to your leaderboard, I wonder would you like to add some system metrics too? From my experience, although some models are not as accurate as the others, they could be significantly faster or less system-demanding. Adding these metrics like training time or memory consumption could also motivate people to develop models that are cheaper and more practical (not always promoting GPT-X style stuffs). What do you guys think? If you think it's a good idea, I'm happy to contribute more thoughts on how to standardize the benchmarking process.

Hi, thank you for your great suggestion. We completely agree that system metric is important; therefore, we have just added two additional fields in our leaderboards: #parameters and hardware used in the experiments (see [here](https://ogb.stanford.edu/docs/leader_overview/)).

Regarding the training time and memory consumption, while we agree that they are relevant metrics, they are also hard to measure in a fair manner due to the difference in the package, computing infrastructure etc. To strike the good balance between simplicity and practical usefulness, we decided to only require the two additional pieces of information.


Cool! The new fields make a lot of sense to me. Keep up the great work!
Repository: snap-stanford/ogb
Issue: unzipping wikikg90m-v2 doesn't work
Root cause: Environment Incompatibility

Hi,

the script successfully downloads the zip file, but it produces an error while trying to unzip it:

>>> from ogb.lsc import WikiKG90Mv2Dataset
>>> dataset = WikiKG90Mv2Dataset(root = '/mnt/data2/ogb-lsc')
This will download 88.24GB. Will you proceed? (y/N)
y
Using exist file wikikg90m-v2.zip
Extracting /mnt/data2/ogb-lsc/wikikg90m-v2.zip
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/lib/python3.7/site-packages/ogb/lsc/wikikg90mv2.py", line 37, in __init__
    self.download()
  File "/opt/conda/lib/python3.7/site-packages/ogb/lsc/wikikg90mv2.py", line 60, in download
    extract_zip(path, self.original_root)
  File "/opt/conda/lib/python3.7/site-packages/ogb/utils/url.py", line 87, in extract_zip
    with zipfile.ZipFile(path, 'r') as f:
  File "/opt/conda/lib/python3.7/zipfile.py", line 1258, in __init__
    self._RealGetContents()
  File "/opt/conda/lib/python3.7/zipfile.py", line 1325, in _RealGetContents
    raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file

Would you mind to check this?

Thank you,
Daniel

It's likely because your downloaded file is corrupted. I suggested you delete the zip file and run your script again to re-download.

Thanks, it solved my problem.
Repository: snap-stanford/ogb
Issue: ImportError: cannot import name 'smiles2graph' from 'ogb.utils'
Root cause: Environment Incompatibility

Hi,

I'm unable to import _smiles2graph_. I've tried installing rdkit and even multiple versions of it but without result. I'm on torch 1.10.2, cuda 11.3, pyg 2.0.4 and Windows 10.

`from ogb.utils import smiles2graph
ImportError: cannot import name 'smiles2graph' from 'ogb.utils'`

You need to [install `rdkit`](https://pypi.org/project/rdkit-pypi/) to use the molecule data. 

Hi, thanks for the reply. As I've said I already did install rdkit(even tried multiple versions). Even so the error persists. Do you have any idea what the problem could be?

Here's an update: if I remove the try-except block in the _utils/__init__.py_ file I get the following error:
```
from rdkit.Chem.rdmolfiles import *
ImportError: DLL load failed while importing rdmolfiles: The specified procedure could not be found.
```

I've tried the command from [this thread](https://github.com/rdkit/rdkit/issues/1989) but with no success.

This seems not related to our package. You should consult with rdkit to install it properly.
Repository: snap-stanford/ogb
Issue: Nan value in the labels of "ogbg-mol-tox21" dataset
Root cause: Documentation Deficiency

Hi OGB Team,

I'm using the `ogbg-mol-tox21` dataset, but it seems there are some missing values in the labels.

```
ipdb> dataset = PygGraphPropPredDataset(name='ogbg-mol-tox21')
ipdb> dataset
PygGraphPropPredDataset(7831)
ipdb> dataset[0]
Data(edge_attr=[34, 3], edge_index=[2, 34], id=[1], x=[16, 9], y=[1, 12])
ipdb> dataset[0].y
tensor([[0., 0., 1., nan, nan, 0., 0., 1., 0., 0., 0., 0.]], dtype=torch.float64)
ipdb> dataset[1].y
tensor([[0., 0., 0., 0., 0., 0., 0., nan, 0., nan, 0., 0.]], dtype=torch.float64)
ipdb> dataset[3].y
tensor([[0., 0., 0., 0., 0., 0., 0., nan, 0., nan, 0., 0.]], dtype=torch.float64)
```
Would you please take a look at this problem?


nan represents "not labeled" (not positive nor negative), which will be ignored during evaluation.