zhiminy/EvalEng huggingface dataset:
Repository: snap-stanford/ogb
Issue: is there any plan to add dynamic or temporal graph dataset for benchmark evaluation?

in this article, the author said "We are now working on creating new dynamic graph datasets and tasks as part of the [Open Graph Benchmark](https://ogb.stanford.edu/docs/team/)."

is there any progress?

Yes, that's ongoing. We will let you know once we reach there (hopefully in the next few months).

Hi @weihua916, I may have just the thing ready for these tasks. The graphs I have [mentioned in this other issue](https://github.com/snap-stanford/ogb/issues/422) are released in permalinks every month or so, so you have this rather large list of evolving graphs with many checkpoints. Several of these graphs have both node and edge labels and have features associated with both nodes, node types and edges.

Possibly they just might do the trick, and since these graphs are actively used in research, getting good predictions would be readily useful to the [team at Berkeley](http://www.berkeleybop.org/) that develops them.

Our TGB paper is online: https://arxiv.org/abs/2307.01026
Repository: snap-stanford/ogb
Issue: Graph reports
Root cause: Unimplemented Feature Gap

Hi!

I have started running some experiments with OGB graphs, and I have noticed some of them include some topological oddities (isomorphic nodes, dendritic trees etc). I am not sure whether these are problematic for the task, as it is generally extremely task-dependent.

That being said, [using GRAPE utilities for graph report](https://github.com/AnacletoLAB/grape/blob/main/tutorials/Create%20extensive%20knowledge%20graph%20reports%20using%20GRAPE.ipynb) I can get you for all graphs rather quickly extensive reports, such as the one shown in the linked example.

These reports are generated in HTML (or Markdown) and can be readily dropped into the OGB website if desired - I believe they may offer insights to OGB users regarding the graph topology, and why their model is performing well / not so well on a given graph.

[Here is another example of a graph report, computed on the PubMed citation graph](https://docs.google.com/document/d/1hL6E4-NFODSQiwN8C0HC626_axnYTgPGgh-n4og_CNc/edit?usp=sharing).

If you believe these may be useful I can start providing them as part of a pull request, just let me know if you believe that to be the case. 

Best,
Luca
Repository: snap-stanford/ogb
Issue: Actual value of the test set
Root cause: Documentation Deficiency

Hello, I am currently working on my graduation thesis and using the dataset pcqm4m-v2. My task is actually similar to the KDDCUP challenge. I have encountered a problem now, which is that if I train the model based on the dataset you divided, I can only calculate the MAE of my model on the validation set. However, I need to know how the MAE is on the test set, which can be found on the dataset you provided, The HOMO-LUMO energy gaps corresponding to the test set are all nan. May I ask if it is convenient for you to provide me with the actual values of the test set?

For the test set, you will need to submit your prediction to our leaderboard.
Repository: snap-stanford/ogb
Issue: ogbg-code: AST and DAGs

In the paper you write that the dataset consists of trees. Is it correct that, by adding the next_token edges (but no inverse edges), we obtain DAGs? Or is it possible that the AST order is different from the token sequence order so that there might be cycles? 

Thank you for taking the benchmarking initiative, the datasets are great!

That's correct -- by adding the `next_token` edges while respecting their directionality (and **not** generating the inverse edges), the original AST will turn into a DAG.  The AST node order does follow the token sequence order.

Thanks for your interest in our datasets.
Looking forward to receiving your submissions!
Repository: snap-stanford/ogb
Issue: Link to download SDF data is down
Root cause: Environment Incompatibility

Hi,

It seems that http://ogb-data.stanford.edu/data/lsc/pcqm4m-v2-train.sdf.tar.gz is down. It is supposed to be the SDF file for the PCQM4Mv2 dataset.

Not sure if it is a temporary outage or if it was accidentally removed.

It was probably temporary. I can download it with
```bash
wget http://ogb-data.stanford.edu/data/lsc/pcqm4m-v2-train.sdf.tar.gz
```
Can you try again?

Right, it works now, thanks.
Repository: snap-stanford/ogb
Issue: how to obtain consistent output under the fixed hyperparameters and seeds?
Root cause: Documentation Deficiency

Hi there, 
I was running your code (https://github.com/snap-stanford/ogb/tree/master/examples/graphproppred/mol) and I noticed that even when I fixed the hyperparameters and seeds, the results vary slightly each time I run the program. Could you please provide some guidance on how to ensure that the program consistently produces the same results under fixed hyperparameters and seeds (in torch, torch.cuda and numpy) ? Thank you very much for your help!

[Here](https://pytorch.org/docs/stable/notes/randomness.html) is how you can fix all seeds.
Repository: snap-stanford/ogb
Issue: Some question in the linkproppred/collab /gnn.py

When loading the data:     dataset = PygLinkPropPredDataset(name='ogbl-collab'); data = dataset[0]; edge_index = data.edge_index; I wonder the edge_index means what? Whether it represents the complete topology or the topology of the training set. If the latter, why is it not the same as split_edge['train']['edge']?

Topology of the training set. It is the same as `split_edge['train']['edge']` except that `data.edge_index` is undirected.
Repository: snap-stanford/ogb
Issue: Raw Text for MAG240M and WikiKG90Mv2
Root cause: Documentation Deficiency

Hello @weihua916,

Do you have the raw text data for these two datasets? Or can you tell me how to get their corresponding text data and how to build the mapping between text data and node ids? I want to get the text data so that I can use other LM to generate better embeddings.

Any help or guidance will be much appreciated.
Best wishes,
Haozhan

Hi Haozhan, thanks for reaching out!

Just released!: https://groups.google.com/g/open-graph-benchmark/c/R0SKtj9qQyE
Repository: snap-stanford/ogb
Issue: Why is the result for GCN so low for ogbn-proteins?

The results for GCNs are suspiciously low, just by examination.

When I run it (with default settings), I got:
```
Highest Train: 82.77 ± 0.21
Highest Valid: 79.02 ± 0.30
  Final Train: 82.72 ± 0.17
   Final Test: 71.96 ± 0.59
```

 Judging from the variance numbers reported, these numbers are very incongruous with the leaderboard result of 65.11 +-1.52

Just a thought, was it perhaps run it with epochs=500? At that point, the default GCN seems to be around 65%.

Thanks for this issue. You are absolutely right. The GCN proteins results are based on an earlier experiment where we only trained for 500 epochs. I retrained the script and it achieves about 72.51 ROCU-AUC on the test set (which is still not that good though). We will update our leaderboards accordingly. Sorry for this mistake.
Repository: snap-stanford/ogb
Issue: dataset not available due to url issue
Root cause: External Dependency Breakage

It seems that  `http://snap.stanford.edu/` site is down and dataset is not available.
Would there be a resolution to this any time soon?

Such as the following dataset are not available as a result.

Thank you,

```
url,http://snap.stanford.edu/ogb/data/nodeproppred/proteins.zip,http://snap.stanford.edu/ogb/data/nodeproppred/products.zip,http://snap.stanford.edu/ogb/data/nodeproppred/arxiv.zip,http://snap.stanford.edu/ogb/data/nodeproppred/mag.zip,http://snap.stanford.edu/ogb/data/nodeproppred/papers100M-bin.zip
```


I have problem downloading data as well. I am trying to download "ogbn-arxiv" dataset for CS224W from this notebook: 

`https://colab.research.google.com/drive/1xHmpjVO-Z74NK-dH3qoUBTf-tKUPfOKW?usp=sharing#scrollTo=Gpc6bTm3GF02`

It seems that all dataset is unavailable.

Failed to download `ogbn-arxiv`: `python3 -c "import dgl;from ogb.nodeproppred import DglNodePropPredDataset; ds = DglNodePropPredDataset('ogbn-arxiv')"`

Same issue. The datasets ogbg-molhiv and ogbg-moltox21 are unavailable.

Same problem. It has "**urllib.error.URLError: <urlopen error [Errno 110] Connection timed out>**" **Error** when downloading dataset from http://snap.stanford.edu/ogb/data/graphproppred/csv_mol_download/hiv.zip

I got the same error. I need to include the results of the evaluation using this dataset in my paper, and the deadline is approaching, so please resolve this issue as soon as possible.

Please resolve this issue as soon as possible, the deadline for my paper is coming!

Thanks for letting us know. This is due to Stanford's maintenance: https://cs.stanford.edu/srcf-shutdown
The server will be ready to use from July 4th again. Apologies for the inconvenience. Thanks for your patience.

Isn't there any other way to download these datasets?

The dataset server is up again. You can download the data now.
Repository: snap-stanford/ogb
Issue: ogb download dataset TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
Root cause: External Dependency Breakage

https://github.com/dmlc/dgl/blob/40dcc7152d930a8761b84d9ed2041a32fa06972c/examples/pytorch/graphsage/node_classification.py#L181


    dataset = AsNodePredDataset(DglNodePropPredDataset("ogbn-products"))
  File "/opt/conda/lib/python3.7/site-packages/ogb/nodeproppred/dataset_dgl.py", line 69, in __init__
    self.pre_process()
  File "/opt/conda/lib/python3.7/site-packages/ogb/nodeproppred/dataset_dgl.py", line 98, in pre_process
    if decide_download(url):
  File "/opt/conda/lib/python3.7/site-packages/ogb/utils/url.py", line 13, in decide_download
    size = int(d.info()["Content-Length"])/GBFACTOR
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'

Maybe it is because URL fail, let's wait until 4th July


I am getting a similar error and suspect the url failure as I tried both the loaders `PygGraphPropPredDataset` and `GraphPropPredDataset` as well specifically for the "ogbg-molhiv" dataset.
What is it about 4th July?
And is there any other way to download the dataset?


> I am getting a similar error and suspect the url failure as I tried both the loaders `PygGraphPropPredDataset` and `GraphPropPredDataset` as well specifically for the "ogbg-molhiv" dataset. What is it about 4th July? And is there any other way to download the dataset?

https://github.com/snap-stanford/ogb/issues/452#issuecomment-1608233455
Repository: snap-stanford/ogb
Issue: AttributeError: 'GlobalStorage' object has no attribute 'adj_t'
Root cause: Documentation Deficiency

I loaded `ogbn-arxiv`, and then trained this dataset with `GCN` model.
I encontered this error, but I donnot know why.
my environment:
```
ogb    1.3.6
python   3.7.16
torch-geometric   2.0.4
```

```
Traceback (most recent call last):
  File "/home/CCSLab/.conda/envs/dxf-GCN-npu-test/lib/python3.7/site-packages/torch_geometric/data/storage.py", line 50, in __getattr__
    return self[key]
  File "/home/CCSLab/.conda/envs/dxf-GCN-npu-test/lib/python3.7/site-packages/torch_geometric/data/storage.py", line 70, in __getitem__
    return self._mapping[key]
KeyError: 'adj_t'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "gcn-modified-npu.py", line 62, in <module>
    train()
  File "gcn-modified-npu.py", line 43, in train
    out = model(data)
  File "/home/CCSLab/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "../model_library/models/GCNNet_ogbn.py", line 32, in forward
    x, adj_t = data.x, data.adj_t
  File "/home/CCSLab/.conda/envs/dxf-GCN-npu-test/lib/python3.7/site-packages/torch_geometric/data/data.py", line 362, in __getattr__
    return getattr(self._store, key)
  File "/home/CCSLab/.conda/envs/dxf-GCN-npu-test/lib/python3.7/site-packages/torch_geometric/data/storage.py", line 53, in __getattr__
    f"'{self.__class__.__name__}' object has no attribute '{key}'")
AttributeError: 'GlobalStorage' object has no attribute 'adj_t'
THPModule_npu_shutdown success.

```

Hi!
It's likely that you did not convert COO `edge_index` into `SparseTensor` format. See [here](https://github.com/snap-stanford/ogb/blob/master/examples/nodeproppred/arxiv/gnn.py#L131-L132). The following should work.
```python
import torch_geometric.transforms as T
from ogb.nodeproppred import PygNodePropPredDataset
dataset = PygNodePropPredDataset(name='ogbn-arxiv', transform=T.ToSparseTensor())
```
Repository: snap-stanford/ogb
Issue: Some questions about `PygNodePropPredDataset()`
Root cause: Documentation Deficiency

I load `ogbn-arxiv` in this way 
```
dataset = PygNodePropPredDataset(name='ogbn-arxiv', root='./arxiv')
```
I want to use dense Tensor, so I haven't assigned the 'transform' argument. But I found that my dataset has been transformed into 'SparseTensor' automatically! But I donnot know why.

Is there any way to use 'ogbn-arxiv' dataset as dense tensor?

Hi! What do you mean by "dense" tensor? You will get the standard COO sparse matrix representation for `dataset[0].edge_index`. See below.

```bash
>>> from ogb.nodeproppred import PygNodePropPredDataset
>>> import torch_geometric
>>> torch_geometric.__version__
'2.4.0'
>>> dataset = PygNodePropPredDataset(name='ogbn-arxiv', root='./arxiv')
Downloading http://snap.stanford.edu/ogb/data/nodeproppred/arxiv.zip
Downloaded 0.08 GB: 100%|█████████████████████████████████████████████| 81/81 [00:05<00:00, 15.14it/s]
Extracting ./arxiv/arxiv.zip
Processing...
Loading necessary files...
This might take a while.
Processing graphs...
100%|████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 59918.63it/s]
Converting graphs into PyG objects...
100%|█████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1067.52it/s]
Saving...
Done!
>>> dataset[0]
Data(num_nodes=169343, edge_index=[2, 1166243], x=[169343, 128], node_year=[169343, 1], y=[169343, 1])
>>> dataset[0].edge_index
tensor([[104447,  15858, 107156,  ...,  45118,  45118,  45118],
        [ 13091,  47283,  69161,  ..., 162473, 162537,  72717]])
```
Repository: snap-stanford/ogb
Issue: Problem on loading "ogbg-moltox21" with virtual node
Root cause: Validation Gap

Hi,

Thanks for providing this very helpful dataset. I got an error on loading the "ogbg-moltox21" dataset with the transform as the virtual node, could you take a look? The error can be reproduced using the following code piece:
`
import torch_geometric.transforms as T
from ogb.graphproppred import PygGraphPropPredDataset
from torch_geometric.loader import DataLoader

name = 'ogbg-moltox21'
dataset = PygGraphPropPredDataset(root="data/"+name, name=name, transform=T.VirtualNode())
loader = DataLoader(dataset, batch_size=256, shuffle=True)
for data in loader:
    print(data.x.shape)
`

and the error information is:
Traceback (most recent call last):
  File "/home/zhexu3/CausalGraphTransformer/test_loader.py", line 8, in <module>
    for data in loader:
  File "/home/zhexu3/ENTER/envs/torch39/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 628, in __next__
    data = self._next_data()
  File "/home/zhexu3/ENTER/envs/torch39/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 671, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/zhexu3/ENTER/envs/torch39/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 58, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/zhexu3/ENTER/envs/torch39/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 58, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/zhexu3/ENTER/envs/torch39/lib/python3.9/site-packages/torch_geometric/data/dataset.py", line 240, in __getitem__
    data = data if self.transform is None else self.transform(data)
  File "/home/zhexu3/ENTER/envs/torch39/lib/python3.9/site-packages/torch_geometric/transforms/virtual_node.py", line 37, in __call__
    new_type = edge_type.new_full((num_nodes, ), int(edge_type.max()) + 1)
RuntimeError: max(): Expected reduction dim to be specified for input.numel() == 0. Specify the reduction dim with the 'dim' argument.

I am using python 3.9, pyg 2.2.0, and ogb 1.3.6. Let me know if you need more information to locate this error. I appreciate it very much!

More updates on this error:
(1) Seems for ogbg-molpcba, the problem exists too, when the batch_size of the dataloader is ~4000-5000.
(2) However, for ogbg-molpcba, when the batch_size of the loader is 256, the problem disappears.

Thanks for reporting. Seems like `edge_type` is empty. Maybe @rusty1s can take a look?
Repository: snap-stanford/ogb
Issue: Faster loading of MAG240M feats in DRAM
Root cause: Unimplemented Feature Gap

Hello Team,

I have sufficient DRAM in my system close to 750G, and am looking to load feats in_memory to exploit faster DRAM access. However I see the format stored is .npy which makes the loading process extremely slow.

Like the ogbn_papers100M and other family of datasets we use .npz compressed format and also store in a preprocessed directory in binary format which makes loading times from disk extremely fast. 

Is it possible to re-use the same libraries for MAG240M datastets, or is there any workaround?

TIA.
Repository: snap-stanford/ogb
Issue: EOFError: Ran out of input
Root cause: Environment Incompatibility

Hi! I get the following error when trying to load the "ogbn-mag" dataset running on Singularity. When I run the same script locally, I don't get the error.
<img width="752" alt="Screenshot 2023-10-02 at 1 41 19 PM" src="https://github.com/snap-stanford/ogb/assets/33410590/b63fd5e8-0abd-43de-ac5b-2adfe9dc68bc">

I also get this error when trying to import via the command line as well (also on Singularity and not locally).
<img width="762" alt="Screenshot 2023-10-02 at 1 42 19 PM" src="https://github.com/snap-stanford/ogb/assets/33410590/b6568f45-67c4-4c9b-be96-aa6f05d979fb">

I'd be happy to provide any more information that's needed. (ie, more information about my setup, the code I'm running, versions of packages, etc) 

Also this doesn't happen with any other datasets. Just the MAG one

Sorry! Figured out the problem immediately after posting. Closing the issue
Repository: snap-stanford/ogb
Issue: OGB papers 100M required Hw

Hi,

In order to run papers 100M node prediction task, what are the CPU Hw requirements?
1. DDR
2. Disk space
3. ...



Most likely more than 100GB CPU memory if you are to read the entire data into memory.

I have 256GB of DDR , but still see OOM 

Interesting.. Depending on the computation you perform, it can certainly require more memory.

I think we should re-open this issue and look at this problem in detail. It doesn't feel right to require more than 256GB of memory to pre-process `ogbn-papers100M`...

To clarify, I am also having this issue simply from loading the dataset (not performing any additional computations).

I run out of memory using 256GB purely by running `dataset = PygNodePropPredDataset('ogbn-papers100M')`.

This has been resolved in `ogb==1.2.3` by using binary data saving/loading.
Repository: snap-stanford/ogb
Issue: It promotes that Segmentation fault (core dumped) when from ogb.nodeproppred import DglNodePropPredDataset
Root cause: External Dependency Breakage

dgl: 0.9.1post1
ogb: 0.9.1post1 
OS: ubuntu 18.04

When executing the cmd: from ogb.nodeproppred import DglNodePropPredDataset, it promotes the error: Segmentation fault (core dumped).

Would you like to help me?
Thanks
Repository: snap-stanford/ogb
Issue: An error occurred while downloading the dataset arxiv！
Root cause: External Dependency Breakage

i follow this example 100% （https://github.com/snap-stanford/ogb/blob/master/examples/nodeproppred/arxiv/gnn.py） and then give me a error   as follow：  
********************************************************
raise BadZipFile("File is not a zip file")   zipfile.BadZipFile: File is not a zip file
*********************************************************
Additionally, I have obtained a folder in the current directory with the following contents
![dir](https://github.com/snap-stanford/ogb/assets/38686239/1d106a5d-1a08-41f5-b6a0-7231b9a05e20)
![url](https://github.com/snap-stanford/ogb/assets/38686239/b228e1e5-53b2-4fd4-99dd-787cb6d0d5ee)

plz help me ，i wanna use this dataset to do some experiment.

Can you delete those files and try again? It's likely that download of zipped folder was not successful.

Same problem during downloading ogbn-products, it seems this issue happens recently, didn't encounter this before. Deleting files doesn't work for me. Any help would be appreciated.


> Same problem during downloading ogbn-products, it seems this issue happens recently, didn't encounter this before. Deleting files doesn't work for me. Any help would be appreciated.

Did you solve it? I met this problem too

@shexuan Can you paste the error message?
Repository: snap-stanford/ogb
Issue: Raw text features for OGBN-MAG dataset
Root cause: Documentation Deficiency

Hi, I was wondering if there is a way to access the raw text features of paper nodes in ogbn-mag dataset. I tried to fetch their metadata from [papers100M dataset](https://github.com/snap-stanford/ogb/tree/master/examples/nodeproppred/papers100M), but only 600 of 73.6K got matched based on entity_id provided in paper_entidx2name.csv. Meanwhile, the MAG service is retired, and we can no longer access the MAG from their [website](https://www.microsoft.com/en-us/research/project/open-academic-graph/). Is there any chance the team still keeps the metadata or any alternative ways to obtain them? Btw, the entity ids of papers provided under the mapping folder correspond to OAG v1 or v2? Is it consistent with the entity name from the papers100M dataset? Thanks!

see examples/llm/glem.py
solves it