zhiminy/EvalEng huggingface dataset:
Repository: snap-stanford/ogb
Issue: homo_data.edge_index has redundant edges for ogbn_mag dataset
Root cause: Validation Gap

Hi OGB Team, hope you are doing great.
For ogbn-mag dataset, I found `homo_data.edge_index` in `cluster_gcn.py` file has many redundant edges with the same edge type. For example, edge (196525, 1199171) occurs many times just like the below screenshot shows. Are there any potential bugs of `group_hetero_graph(data.edge_index_dict, data.num_nodes_dict)` statement? Thank you for answering in advance!
<img width="861" alt="Screen Shot 1441-11-04 at 6 42 38 PM" src="https://user-images.githubusercontent.com/29864861/85857305-42c58200-b7c2-11ea-8aa7-f5f48ea6a2c2.png">


Hi, thanks for noticing it. The bug is not on PyG side, but on the OGB dataset side. At the pre-processing stage, we mistakingly included the duplicated edges. We have fixed the issue (removed the duplicated edges) on our side. Fortunately, we found the empirical results are almost unaffected.

We will update the dataset, package, and the leaderboard as soon as possible, possibly by this weekend.

Thanks,
OGB Team

Hi OGB Team, thanks for replying. Cool :)

Updated the package :)
Repository: snap-stanford/ogb
Issue: [Question] Why are not all MoleculeNet datasets implemented?

A few large datasets from MoleculeNet, concerning quantum chemistry, are not implemented in OGB, and also PDBbind dataset. What is the reason for this? I understand that QM datasets typically use specific features, but e.g. [this paper](https://pubs.acs.org/doi/full/10.1021/acs.jcim.9b00237) use regular features (like Morgan / ECFP / circular fingerprints) and get good results. I think that they could be added to OGB with regular set of features. They can also use scaffold split, like in paper I linked before, similar to other MoleculeNet dataset.

We could certainly include it, but now we have a better one called [PCQM4Mv2](https://ogb.stanford.edu/docs/lsc/pcqm4mv2/).

In my opinion, QM9 is not so realistic and somewhat solved (SoTA MAE already below the chemical accuracy). May be a bit outdated to include by now.
Repository: snap-stanford/ogb
Issue: Python3.9:  OSError: libcublas.so.11: cannot open shared object file: No such file or directory
Root cause: Environment Incompatibility

Aware, of course, that you don't yet officially support Python3.9, but thought this might be useful for someone who needs to use it:

Had this error when trying to install the module on Python3.9:

```
/python3.9/site-packages/torch/__init__.py", line 153, in _load_global_deps
     ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
   File "/usr/lib64/python3.9/ctypes/__init__.py", line 374, in __init__
     self._handle = _dlopen(self._name, mode)
 OSError: libcublas.so.11: cannot open shared object file: No such file or directory
```
Worked around the error by updating the torch version in our model's `requirements.txt` to include a different cuda version (when running on Python3.9 or greater) thus:

```
# Workaround for OGB issue on Python3.9
--find-links https://download.pytorch.org/whl/torch_stable.html
torch==1.13.0+cu116; python_version >= '3.9'
```
Repository: snap-stanford/ogb
Issue: Some problem about loading ogbn-papers100M
Root cause: Documentation Deficiency

Hi! I'm trying to load the ogbn-papers100M with following code:
```
from ogb.nodeproppred import NodePropPredDataset
dataset = NodePropPredDataset(name='ogbn-papers100M')
```
but it output some errors like (part of it):
```
Traceback (most recent call last):
  File "node_prop_pred_data.py", line 24, in <module>
    dataset = NodePropPredDataset(name=dataset_name, root = dataset_save_location)
  File "/home/min/a/user/data/.venv/lib/python3.6/site-packages/ogb/nodeproppred/dataset.py", line 63, in __init__
    self.pre_process()
  File "/home/min/a/user/data/.venv/lib/python3.6/site-packages/ogb/nodeproppred/dataset.py", line 139, in pre_process
    torch.save({'graph': self.graph, 'labels': self.labels}, pre_processed_file_path, pickle_protocol=4)
  File "/home/min/a/user/data/.venv/lib/python3.6/site-packages/torch/serialization.py", line 372, in save
    _save(obj, opened_zipfile, pickle_module, pickle_protocol)
MemoryError -- std::bad_alloc
```
It seems like `OOM`...
The memory size of my machine is 256GB

BTW, I also try to load the raw graph by myself. But when I load the node_label.npz :
```
>>> import numpy as np
>>> label = np.load('node-label.npz')
>>> label
<numpy.lib.npyio.NpzFile object at 0x2b15ac5232b0>
>>> label.__dict__
{'_files': ['node_label.npy'], 'files': ['node_label'], 'allow_pickle': False, 'pickle_kwargs': {'encoding': 'ASCII', 'fix_imports': True}, 'zip': <zipfile.ZipFile file=<_io.BufferedReader name='node-label.npz'> mode='r'>, 'f': <numpy.lib.npyio.BagObj object at 0x2b15df759640>, 'fid': <_io.BufferedReader name='node-label.npz'>}
>>> node_label = label['node_label']
>>> node_label
array([[ nan],
       [ nan],
       [ nan],
       ...,
       [157.],
       [ nan],
       [ nan]], dtype=float32)
>>>
```

Why there are so many nan on the label file?

I think the error is due to the corrupted file. You may delete the old file and download it again.
The `nan` value means **unlabeled**.

@zhuangbility111 I did not receive out of memory but get the same `nan` when I run `np.load('node-label.npz')`. Have you fixed the issue?

Got it. To whom see the issue, it is due to the paper100M has unlabeled nodes. `nan` is normal.
Repository: snap-stanford/ogb
Issue: Bug: root path can not use `~` to indicate home directory
Root cause: Validation Gap

Hi @weihua916 , I have just find a bug when I use `~/public_data`  as the root directory, it will crashed and raise some errors below:
```
Downloading http://snap.stanford.edu/ogb/data/nodeproppred/mag.zip
Traceback (most recent call last):
  File "/home/hzcheney/miniconda3/envs/maskgae/lib/python3.8/site-packages/ogb/utils/url.py", line 59, in download_url
    with open(path, 'wb') as f:
FileNotFoundError: [Errno 2] No such file or directory: '~/public_data/pyg_data/mag.zip'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train_nodeclas.py", line 242, in <module>
    dataset = PygNodePropPredDataset(root=root, name=f'ogbn-{args.dataset}')
  File "/home/hzcheney/miniconda3/envs/maskgae/lib/python3.8/site-packages/ogb/nodeproppred/dataset_pyg.py", line 68, in __init__
    super(PygNodePropPredDataset, self).__init__(self.root, transform, pre_transform)
  File "/home/hzcheney/miniconda3/envs/maskgae/lib/python3.8/site-packages/torch_geometric/data/in_memory_dataset.py", line 56, in __init__
    super().__init__(root, transform, pre_transform, pre_filter)
  File "/home/hzcheney/miniconda3/envs/maskgae/lib/python3.8/site-packages/torch_geometric/data/dataset.py", line 84, in __init__
    self._download()
  File "/home/hzcheney/miniconda3/envs/maskgae/lib/python3.8/site-packages/torch_geometric/data/dataset.py", line 145, in _download
    self.download()
  File "/home/hzcheney/miniconda3/envs/maskgae/lib/python3.8/site-packages/ogb/nodeproppred/dataset_pyg.py", line 126, in download
    path = download_url(url, self.original_root)
  File "/home/hzcheney/miniconda3/envs/maskgae/lib/python3.8/site-packages/ogb/utils/url.py", line 69, in download_url
    raise RuntimeError('Stopped downloading due to interruption.')
RuntimeError: Stopped downloading due to interruption.

```
After I change the root directory to `/home/hzcheney/public_data`, it will work as usual.
- python 3.8.15
- pytorch 1.12.1
- torch_geometric 2.0.4
- ogb 1.3.4

Hi! What's the output of `echo $HOME` in your environment?
Repository: snap-stanford/ogb
Issue: Redundant self connections in ogbn-products 
Root cause: Validation Gap

Dear OGB Team,

Thanks for providing these awesome datasets!

I noticed that for the ogbn-products dataset, there are some redundant self connections. For example, the edge (15064,15064) occurred twice in the edge_index array. 
![2](https://user-images.githubusercontent.com/6771308/86067451-bf0dce80-ba29-11ea-9a4c-e82f02d43d98.png)

Here is a list of the row and column indices of the repeated edges. After removing the repeated edges, it gives me 123718152 (doubled) edges instead of the original 123718280 edges.
![1](https://user-images.githubusercontent.com/6771308/86067447-be753800-ba29-11ea-9595-917c54125b22.png)

Best,
Tedzhouhk



Thank you very much for pointing this out. Indeed, a very small number of self-edges do exist in the raw data (which we did not expect), and the issue here is that we have added inverse edges for all the edges (including the self-edges) when making the graph undirected. Given that self-edges are very rare (128 nodes out of 2,449,029 nodes), we are not planning to release a modified dataset, but it is good to keep this in mind. Please feel free to remove the duplicated self-edges.

We have added a brief note about this issue on our [webpage](https://ogb.stanford.edu/docs/nodeprop/#ogbn-products). Thanks again for noticing this!

Thank you so much for your quick response!
Repository: snap-stanford/ogb
Issue: Cannot load data
Root cause: Validation Gap

I try to load ogbn products
```python
    from ogb.nodeproppred import PygNodePropPredDataset
    root = "/data/"
    dataset = PygNodePropPredDataset(name = "ogbn-products",root=root)
    print(dataset)
    data = dataset[0]
```


It shows 
```
Downloading http://snap.stanford.edu/ogb/data/nodeproppred/products.zip
Downloaded 1.38 GB: 100%|█████████████████████████████████| 1414/1414 [11:18<00:00,  2.09it/s]
Extracting /data/products.zip
Processing...
Loading necessary files...
This might take a while.
Processing graphs...
100%|███████████████████████████████████████████████████| 1/1 [00:03<00:00,  3.87s/it]
Converting graphs into PyG objects...
100%|█████████████████████████████████████████████████| 1/1 [00:00<00:00, 3144.16it/s]
Saving...
Done!


PygNodePropPredDataset()
Traceback (most recent call last):
  File "microbatch__quiver_base.py", line 152, in <module>
    train(data, quiver_feature, quiver_sampler, dataset, args)
  File "microbatch__quiver_base.py", line 26, in train
    if args.dataset == 'ogbn-products':
AttributeError: 'Namespace' object has no attribute 'dataset'
```
Repository: snap-stanford/ogb
Issue: Atom Feature for Non-Tetrahedral Stereochemistry
Root cause: Validation Gap

With this https://github.com/rdkit/rdkit/pull/5084 being introduced, the `atom.GetChiralTag()` call can return additional chiralities like
```
CHI_OCTAHEDRAL
CHI_TRIGONALBIPYRAMIDAL
CHI_SQUAREPLANAR
CHI_ALLENE
```
This gives an error if I use the `ogb.atom_to_feature_vector` call. I am not sure what other places in the code will be affected by this, but I see an addition to this [list](https://github.com/snap-stanford/ogb/blob/d37cffa2e2cde531ca7b7e75800d331ed1e738a6/ogb/utils/features.py#L4) being one of the changes. 

I've opened a PR with suggested changes. Feel free to continue the PR.

Thanks for the PR. While it is nice to introduce advanced features, I also want to maintain backward compatibility with existing models. I think it makes sense to include these features once they are proven useful for ML tasks. What do you think? Did you find these features actually influence/improve ML model performance?

> I also want to maintain backward compatibility with existing models

I think the thing that would change in terms of backward compatibility is the scope of `msc`. The `idx` for         `'CHI_UNSPECIFIED', 'CHI_TETRAHEDRAL_CW', 'CHI_TETRAHEDRAL_CCW', 'CHI_OTHER',` would be the same as before.

>Did you find these features actually influence/improve ML model performance?

I actually faced this issue when I was trying to set features for [Molecule3D](https://github.com/divelab/MoleculeX/tree/molx/Molecule3D). Not so sure how much this would affect performance.
Repository: snap-stanford/ogb
Issue: Some question about ogbn-mag NeighborSampling (R-GCN aggr)

[https://github.com/snap-stanford/ogb/blob/master/examples/nodeproppred/mag/sampler.py](url)
When I proflied this program with Nsight, I found that many Memcpy DtoH operations appeared dispersedly in the forward and backward stages of training. There is a small gap behind Memcpy DtoH. I want to know the reason why Memcpy DtoH appears, because the data copy from D to H in the compute process is very strange. In addition, these Memcpy DtoH appear behind DeviceReduceKernel and DeviceReduceSingleTileKernel. These operations also appear in a Linear. I also want to know what is the connection between Memcpy DtoH and DeviceReduceSingleTileKernel, and why these two operations always appear together？

This is best asked in torch geometric repo.
Repository: snap-stanford/ogb
Issue: Raw Text for ogbn-products
Root cause: Documentation Deficiency

Hello @weihua916,

Do you, by any chance, have the raw text Amazon dataset from which ogbn-products was generated? In particular, I'm interested to map graph node features to text descriptions and labels to product text labels. I'm looking for something similar to the raw text files you provide for ogbn-arxiv and ogbn-papers100M. I did not find it in the dataset linked in Cluster-GCN. I also tried to download Amazon3M from [here](http://manikvarma.org/downloads/XC/XMLRepository.html), but I could not establish the  mapping.

Any help or guidance will be much appreciated.
Best regards,
Hani

Hi Hani, thanks for reaching out. I do not have raw text but we provided the `mapping` directory that contains a file that maps each node into ASIN (amazon' product's unique ID). It can be used together with the Cluster-GCN's Amazon3M dataset to establish the mapping. Let me know if you have further questions.

Hi, weihua916. I have found an empty label in the labelidx2productcategory.csv.gz and a label `#508510`. What do these two labels mean?
<img width="1497" alt="截屏2024-02-28 19 24 03" src="https://github.com/snap-stanford/ogb/assets/74809554/58905ed5-2a14-4beb-8da1-39009726f80e">
Repository: snap-stanford/ogb
Issue: Question: Confused about NeighborSampler in nodeproppred task

Does NeighborSampler sample duplicate nodes in different batches? If so, maybe some nodes will get more than one result or some nodes never be sampled?

For example, in [examples/products/graph_saint.py](https://github.com/snap-stanford/ogb/tree/master/examples/nodeproppred/products/graph_saint.py), when testing, the model uses a subgraph loader (NeighborSampler) to get testing data.

Subgraph loader is:
https://github.com/snap-stanford/ogb/blob/d37cffa2e2cde531ca7b7e75800d331ed1e738a6/examples/nodeproppred/products/graph_saint.py#L167-L168

Testing code of model:
https://github.com/snap-stanford/ogb/blob/d37cffa2e2cde531ca7b7e75800d331ed1e738a6/examples/nodeproppred/products/graph_saint.py#L41-L58

Yes, for GraphSAINT, there exists no notion of an epoch. We randomly sample nodes for every optimization step. For NeighborSampler and Cluster-GCN, every seed node is only sampled once per epoch.

> Yes, for GraphSAINT, there exists no notion of an epoch. We randomly sample nodes for every optimization step. For NeighborSampler and Cluster-GCN, every seed node is only sampled once per epoch.

Thanks.
Repository: snap-stanford/ogb
Issue: Standard edge-label prediction datasets
Root cause: Unimplemented Feature Gap

Hi all,

I am looking for standard edge-label prediction datasets to evaluate a new model.

If I am not mistaken, this task is not yet covered by OGB.

The [BBOP](http://www.berkeleybop.org/) team has developed [several KGs and ontologies](https://kghub.org/) which may be used for such a task, as most of them have edge labels.

Currently, [800 different KGs versions are available from KGOBO and KGHUB, with 211 unique graphs](https://github.com/AnacletoLAB/grape/blob/main/tutorials/Ensmallen_Automatic_Graph_Retieval_KGOBO.ipynb). I hope that among these we could possibly find a few that are compatible with your data quality standards.

A knowledge graph I believe may be up to this task is [KGCOVID19](http://dx.doi.org/10.1016/j.patter.2020.100155), which you can readily retrieve by running (after installing grape with `pip install grape`):

```python
from grape.datasets.kghub import KGCOVID19

kg = KGCOVID19()
```

This will [download the edge list from kghub](https://kg-hub.berkeleybop.io/kg-covid-19/index.html). I suggest using the aforementioned grape library so you will get a [comprehensive graph report](https://github.com/AnacletoLAB/grape/blob/main/tutorials/Create%20extensive%20knowledge%20graph%20reports%20using%20GRAPE.ipynb) to better evaluate whether this (or other) available KGs may be fit as a standard edge-label prediction benchmark dataset.

I hope it will be possible to readily agree on some standard dataset for edge-label prediction.

I haven't found a discord server for OGB, but if you'd like to quickly talk about this I am available on [GRAPE's discord server](https://discord.gg/Nda2cqYvTN).

Best,
Luca

Hi Luca,

Thanks for reaching out! You are right that we do not have edge-label prediction datasets in OGB, so we welcome dataset contribution here! [Here](https://docs.google.com/document/d/e/2PACX-1vS1hBTYLONRwAU9UxK42USTuRKrt_Yk4H0EhpLvJC_eOrGxbJUtrzDqlIStAFpnwZt2N28B3MuSxgqj/pub) is the instruction how you can contribute some of the datasets! Let us know if you have any further questions.

Best,
Weihua
Repository: snap-stanford/ogb
Issue: Can't load ogbn-arxiv dataset
Root cause: External Dependency Breakage

Hi,

I reinstalled my environment and found I can't load the ogbn-arxiv dataset. Could you please help me solve this issue? 


Python: 3.9
Torch: 2.0.0 with cuda 11.8
Pytorch-Geometric: 2.3.0
ogb: 1.3.5

```
dataset = PygNodePropPredDataset(name='ogbn-arxiv')
Processing...
AttributeError                            Traceback (most recent call last)
Cell In[2], line 1
----> 1 dataset = PygNodePropPredDataset(name='ogbn-arxiv')

File /lib/python3.9/site-packages/ogb/nodeproppred/dataset_pyg.py:68, in PygNodePropPredDataset.__init__(self, name, root, transform, pre_transform, meta_dict)
     65 self.is_hetero = self.meta_info['is hetero'] == 'True'
     66 self.binary = self.meta_info['binary'] == 'True'
---> 68 super(PygNodePropPredDataset, self).__init__(self.root, transform, pre_transform)
     69 self.data, self.slices = torch.load(self.processed_paths[0])

File /lib/python3.9/site-packages/torch_geometric/data/in_memory_dataset.py:57, in InMemoryDataset.__init__(self, root, transform, pre_transform, pre_filter, log)
     49 def __init__(
     50     self,
     51     root: Optional[str] = None,
   (...)
     55     log: bool = True,
     56 ):
---> 57     super().__init__(root, transform, pre_transform, pre_filter, log)
     58     self._data = None
     59     self.slices = None

File lib/python3.9/site-packages/torch_geometric/data/dataset.py:97, in Dataset.__init__(self, root, transform, pre_transform, pre_filter, log)
     94     self._download()
     96 if self.has_process:
---> 97     self._process()

File lib/python3.9/site-packages/torch_geometric/data/dataset.py:230, in Dataset._process(self)
    227     print('Processing...', file=sys.stderr)
    229 makedirs(self.processed_dir)
--> 230 self.process()
    232 path = osp.join(self.processed_dir, 'pre_transform.pt')
    233 torch.save(_repr(self.pre_transform), path)

File lib/python3.9/site-packages/ogb/nodeproppred/dataset_pyg.py:147, in PygNodePropPredDataset.process(self)
    145     additional_edge_files = []
    146 else:
--> 147     additional_edge_files = self.meta_info['additional edge files'].split(',')
    149 if self.is_hetero:
    150     data = read_heterograph_pyg(self.raw_dir, add_inverse_edge = add_inverse_edge, additional_node_files = additional_node_files, additional_edge_files = additional_edge_files, binary=self.binary)[0]

AttributeError: 'float' object has no attribute 'split'
```

Hi! Thanks for reporting. The issue was due to pandas 2.0 update. Can you update ogb package to 1.3.6 and try again?

It works! Thank you Weihua.

Best,
Haoyu

> Hi! Thanks for reporting. The issue was due to pandas 2.0 update. Can you update ogb package to 1.3.6 and try again?

thank u sososososo much you are saving my life buddy!!!!!!!!!
Repository: snap-stanford/ogb
Issue: When ogb is improted into the program, the program does not exit properly
Root cause: Resource Mishandling

Minimum reproducible program：
`import ogb
print(ogb.__version__)`

Problem Cause:
I tried to use from ogb.graphproppred import PygGraphPropPredDataset in my project and found that the program does not exit properly. I tried a simple test program, similar to the code in the minimally reproducible problem, and found that referencing ogb triggered the problem of the program not exiting properly.

I tried copying the relevant implementation from ogb locally and referencing it from a local package and found that the local package does not cause this problem. The only difference between the local and ogb implementations is that there is no `if __name__ == '__main__'` added to the code.

Hi! Thanks for reporting. Could you tell us a bit more about your environment? I cannot reproduce your issue. I can run `import ogb; print(ogb.__version__)` without any problem.

> 

Python version 3.10.0, ogb version 1.3.5, Ubuntu 20.04.6 LTS.

https://github.com/snap-stanford/ogb/issues/408#issuecomment-3632432333

I am able to reproduce this. Perhaps the pyproject.toml should help (I am using uv)
Repository: snap-stanford/ogb
Issue: [Request] Please remove the warning for older versions of OGB
Root cause: Unimplemented Feature Gap

Dear OGB team,

Would it be possible to remove the following warning

    WARNING The OGB package is out of date. Your version is 1.3.5, while the latest version is 1.3.6

from future OGB versions? It affects CLI/GUI based programs that import your package.
Or, is there a way to disable this specific warning?

Many thanks!

Hi thanks for reaching out. Hmm why don't you update your version? As the dataset package, it is important for us to make sure everyone is using the up-to-date package.

Thanks for the prompt answer. Do you mean that you want to be sure everyone can access the latest benchmarks you add?

If that is the only reason, and new releases do not impact the reproducibility of previously added benchmarks, then imho it is a bit of an overkill to add such a message. The user could just update the package and get access to the new features without being reminded every time.

I think the same goes when some main change is introduced: inviting (or urging) users to update the package with that warning could result in irreproducible experiments if subtle changes to the evaluation code were made, and so it would be better to safely upgrade in a second moment, having the user evaluate the changes you introduced.

Maybe you could instead use a `changelog.md` file so that users can readily see if something of their interest changed? This is clearly not a critical issue, but depending on the rate of updates it can become annoying for the user, especially those using CLIs. I have never seen such update warnings being used in other pip packages, and it is always good to stay up to date, but I would not force the users' hand by reminding them every time. This is just my perspective, but I hope you will take this into consideration for future releases.

I am closing the issue since I will leave the final decision to you. I can always disable the warning directly from the package source. But again, I think it is a bit too much for the user.