zhiminy/EvalEng huggingface dataset:
Repository: snap-stanford/ogb
Issue: LSC PCQM4M Dataloader fails with PyTorch < 1.6.0
Root cause: External Dependency Breakage

The LSC dataset generators require the PyTorch version to be => 1.6.0

The following line: 

https://github.com/snap-stanford/ogb/blob/9c28d0e75877368d6a6cb989ba97914e2d2747f9/ogb/lsc/pcqm4m.py#L115

Causes the error: 

```
RuntimeError: split_dict.pt is a zip archive (did you mean to use torch.jit.load()?)
```
The support for zipfile serialization format was introduced in PyTorch 1.6.0 release ([#39460](https://github.com/pytorch/pytorch/pull/39460), [#39893](https://github.com/pytorch/pytorch/pull/39460), [#40288](https://github.com/pytorch/pytorch/pull/40288), [#40793](https://github.com/pytorch/pytorch/pull/40793))

Thank you for noticing it. I have updated the package requirement in the master. This will be reflected in our next package release.

For the time being, let's keep this issue open so that people can easily see it.

No problem. Thanks for the quick fix!
Repository: snap-stanford/ogb
Issue: It would be helpful if you provide the md5 of the zip dataset files
Root cause: Documentation Deficiency

It would be helpful if you provide the md5 of the zip dataset files, as the dataset files are extremely large.

Sure, you can see below (running md5sum over the zipped files), although our package will most likely make sure the downloaded files are correct.

bd61c9446f557fbe4430d9a7ce108b34  mag240m_kddcup2021.zip
73d4f5dde29d78669330b4db4c12fc9c  wikikg90m_kddcup2021.zip
5144ebaa7c67d24da1a2acbe41f57f6a  pcqm4m_kddcup2021.zip

thanks for your response
Repository: snap-stanford/ogb
Issue: PygPCQM4MDataset vs PCQM4MDataset
Root cause: Documentation Deficiency

Hi,

I just wonder why PygPCQM4MDataset does not check "pre_processed_file_path"  like in PCQM4MDataset? 

Thanks 
Bests
ZP



It's because Pytorch Geometric's [InMemoryDataset](https://pytorch-geometric.readthedocs.io/en/latest/modules/data.html#torch_geometric.data.InMemoryDataset) already handles the checking.
Repository: snap-stanford/ogb
Issue: Training time on PCQM4M-LSC

Hi,

Thanks for presenting this great dataset and challenge.

Could you provide the rough training time of baselines (e.g., GCN and GIN) on  PCQM4M-LSC dataset? 

Thank you in advance.

Hi, it takes 1 to 1.5 days on a single GPU.

Hi,

Thank you for the immediate response.

Do you mean the time of training 100 epochs on whole training set? (I guess this is similar to training 1000 epochs on 10% training set.)

It is time for training 100 epochs.

Got it. Thank you for the clarification.
Repository: snap-stanford/ogb
Issue: Why Pre-compute GCN normalization is not used on ogbn-arxiv datasets?
Root cause: Documentation Deficiency

In the folder nodeproppred, all the adj_t have been pre-computed except ogbn-arxiv when selected model is GCN. The default GCNConv can do the same thing, why do we pre-compute? Thanks a lot.

We pre-compute the normalization coefficients of GCN to be a bit more efficient w.r.t. to runtime and memory, which is especially useful on those larger datasets. You can do the same for `ogbn-arxiv` as well, although it is not necessary to fit the GNN on the GPU.
Repository: snap-stanford/ogb
Issue: Could you provide the original HOMO and LUMO energy in PCQM4M-LSC?
Root cause: Unimplemented Feature Gap

Hi,

I found that there is only the HOMO-LUMO gap energy in the dataset. Could you provide the original HOMO and LUMO energy in PCQM4M-LSC?

Thank you in advance.

Hi, during the competition, we do not provide those values. We can provide all the information (including the other quantum values) after the contest.

Okay. Thank you for your response.

Thanks for understanding. We are trying to strike a good balance between practical usefulness and generality and simplicity of ML task.
Repository: snap-stanford/ogb
Issue: PCQM4M-LSC AttributeError: 'NoneType' object has no attribute 'GetAtoms'
Root cause: Environment Incompatibility

Hi,

``` python
from ogb.utils import smiles2graph

# if you use Pytorch Geometric (requires torch_geometric to be installed)
from ogb.lsc import PygPCQM4MDataset
pyg_dataset = PygPCQM4MDataset(root = ROOT, smiles2graph = smiles2graph)
```
The error item_id is 1868736   ('C123C45C61C24C356', 2.5551490561950008)

``` python
graph_obj = smiles2graph('C123C45C61C24C356')
```
At first I thought the downloaded file was corrupted, but after repeating it several times there was still an error. Thank you in advance.

Hi!

It works fine with my environment. My rdkit version is `2019.03.1`. Can you try the same (or newer) rdkit version?

```python
>>> from ogb.utils import smiles2graph
>>> graph_obj = smiles2graph('C123C45C61C24C356')
[08:07:59] WARNING: could not find number of expected rings. Switching to an approximate ring finding algorithm.
>>> graph_obj
{'edge_index': array([[0, 1, 1, 2, 2, 3, 3, 4, 2, 0, 3, 0, 4, 0, 3, 1, 4, 1, 4, 2],
       [1, 0, 2, 1, 3, 2, 4, 3, 0, 2, 0, 3, 0, 4, 1, 3, 1, 4, 2, 4]]), 'edge_feat': array([[0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0]]), 'node_feat': array([[5, 0, 4, 5, 0, 0, 2, 0, 0],
       [5, 0, 4, 5, 0, 0, 2, 0, 0],
       [5, 0, 4, 5, 0, 0, 2, 0, 0],
       [5, 0, 4, 5, 0, 0, 2, 0, 0],
       [5, 0, 4, 5, 0, 0, 2, 0, 0]]), 'num_nodes': 5}
```

Thank you for your generous help! Updating rdkit works for me.
Repository: snap-stanford/ogb
Issue: Unable to download and access data.
Root cause: Environment Incompatibility

```
$curl http://ogb-data.stanford.edu/data/lsc/mag240m_kddcup2021.zip -O mag240m_kddcup2021.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:02:07 --:--:--     0
curl: (7) Failed connect to ogb-data.stanford.edu:80; Connection timed out
curl: (6) Could not resolve host: mag240m_kddcup2021.zip; Name or service not known
```
Unable to download the zip file from several network environments. It seems that the host is down ? 


Indeed there seems to be an on-going issue with downloading. We are looking into it and will get back to you. Thanks for your patience.

It should be fixed now.

Seems to work. Closing the issue. Thanks.
Repository: snap-stanford/ogb
Issue: OGB Datasaver issues
Root cause: Documentation Deficiency

Hi ogb-team!

I followed your tutorial in ogb/ogb/io/Readme.md to create my own custom dataset for a link prediction task.
The tutorial/steps are very straight forward, thank you very much for providing the details and examples.

After that, I tried to include my custom dataset (without official submission) by adding it to the master.csv in the linkpropred via make_master_file.py. So now, when I load my custom dataset via the official API calls, the ogb package fetches my zipped file (via http) and extracts it. However, I get several missing file errors.

Ihave noticed that the storage formats, e.g. ogbl-ddi and ogbl-mydataset are not compatible. Whereas ogbl-ddi (and others) use compressed csv files, the dataset I have obtained in your tutorial uses npy files (in a slightly different folder structure). 
Naturally, the formats can be easily converted to/from, that should not be an issue. However, it would be even more convenient if this step could be omitted. What are your thoughts on that? Please let me know, if you would like someone to implement this.

Hi! Thanks for your question! That's a great point. We have two ways for storing graph data: csv and npy. For the dataset saver, we are using npy, since we later found that it leads to faster and memory-efficient reading/writing compared to the csv format. For the current OGB datasets, only `ogbn-papers100M` is saved using npy format, since this dataset is super huge and using npy format is critical. Other datasets are still saved with the old csv format.

In order for the package to know which format (csv or npy) is used, you need to pass `binary` in your `make_master_file.py`. In your case, since DatasetSaver is using npy format, you need to pass `binary=True`. An example is [here](https://github.com/snap-stanford/ogb/blob/master/ogb/nodeproppred/make_master_file.py#L83). 

Hope this helps!

@weihua916 Thank you very much, that's very helpful!
Repository: snap-stanford/ogb
Issue: label_prop.py fails on MAG240M dataset
Root cause: Validation Gap

https://github.com/snap-stanford/ogb/blob/master/examples/lsc/mag240m/label_prop.py
When I run the above program, I get the following error.
```
Traceback (most recent call last):
  File "label_prop.py", line 52, in <module>
    y_pred = model(y_train, adj_t, train_idx).argmax(dim=-1)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch_geometric/nn/models/label_prop.py", line 46, in forward
    out[mask] = y[mask]
IndexError: index 3739065 is out of bounds for dimension 0 with size 1112392
```

I think it is because the range of values in train_idx, passed to LabelPropagation() as a mask, is up to about 100 million, even though the size of y_train is only about 1.11 million.
I would like to know if this is a problem with my environment or if there is a way to fix it.

My environment is as follows.
ogb==1.3.0
torch==1.8.0
torch-geometric @ git+https://github.com/rusty1s/pytorch_geometric.git@ed97ab797182fb39ab59c6736dbb74a5bdcd41cd

Thanks for reporting. You are absolutely right. I fixed the `label_prop.py ` example in master, see [here](https://github.com/snap-stanford/ogb/blob/master/examples/lsc/mag240m/label_prop.py#L53-L55). Sorry for the inconvenience! Please let me know if everything works as expected.

Thank you, it works!
The following two corrections were necessary. They are really minor fixes.

* add ` import torch.nn.functional as F`
* replace `y_pred = model(y_train, adj_t).argmax(dim=-1)` with `y_pred = model(y, adj_t).argmax(dim=-1)`
Repository: snap-stanford/ogb
Issue: Support for single GPU and single Process training on wikikg90m.
Root cause: Validation Gap

Thanks for your baseline code. It seems that the baseline code does not support single GPU and single Process training on wikikg90m.

I tried to use the following training script to train the model on a single GPU with a single process by specifying `--gpu 0` and `--num_proc 1`.
```bash
CUDA_VISIBLE_DEVICES=0 dglke_train \
--model_name ComplEx \
--hidden_dim 100 --gamma 3 --lr 0.1 --regularization_coef 1e-9 \
--valid --test -adv --mix_cpu_gpu --num_proc 1 --num_thread 1 \
--gpu 0 \
--async_update --force_sync_interval 50000 --no_save_emb \
--print_on_screen --encoder_model_name concat -de -dr --save_path ./save_path
```
However, the baseline code makes the following assertion, and `args.strict_rel_part` must be `true` if we specify `--gpu 0` and `--num_proc 1`
 
https://github.com/snap-stanford/ogb/blob/2229cb40781a8bb3505f0f379869d43e9a74beb9/examples/lsc/wikikg90m/dgl-ke-ogb-lsc/python/dglke/models/general_models.py#L262

The reasons are as follows.

The variable `strict_rel_part` is defined as follows.

https://github.com/snap-stanford/ogb/blob/2229cb40781a8bb3505f0f379869d43e9a74beb9/examples/lsc/wikikg90m/dgl-ke-ogb-lsc/python/dglke/train.py#L152

`args.mix_cpu_gpu` must be set to `true` in order to support the large number of entities. 

`cross_part` is defined as follows. 

https://github.com/snap-stanford/ogb/blob/2229cb40781a8bb3505f0f379869d43e9a74beb9/examples/lsc/wikikg90m/dgl-ke-ogb-lsc/python/dglke/dataloader/sampler.py#L333-L343

If we set `--num_proc 1`, which is the variable `ranks` above, then `cross_part` must be `false` 

My question is, did I miss something or the baseline code actually does not support training with single GPU and single Process?


Hi, one simple fix is to set `self.cross_part = True` on line 342 of the `sampler.py` so that `strict_rel_part` is `False`. Because either way since you only train the model on one gpu and one process, you do not need to partition the relations or create the `global_relation_emb`. At the meantime, we are also doing some test runs for this fix, and will push the fix once we finish the tests. Thank you.

Hi, we have pushed the fix. Now you can run the code on single GPU, and an example is shown below. Note that we do not need the `async_update` flag since we only have one process.

```
CUDA_VISIBLE_DEVICES=0 dglke_train \
--model_name ComplEx \
--hidden_dim 100 --gamma 3 --lr 0.1 --regularization_coef 1e-9 \
--valid --test -adv --mix_cpu_gpu --num_proc 1 --num_thread 1 \
--gpu 0 \
--no_save_emb \
--print_on_screen --encoder_model_name concat -de -dr --save_path ./save_path
```

Thanks for your kind response!
Repository: snap-stanford/ogb
Issue: GPU memory leakage in the baseline code of wikikg90m
Root cause: Resource Mishandling

Hi, thanks for your code.

I noticed that there is a GPU memory leakage problem in the baseline code of wikikg90m.

https://github.com/snap-stanford/ogb/blob/4db49c17a0c51352990b6468803ef5eb495817dd/examples/lsc/wikikg90m/dgl-ke-ogb-lsc/python/dglke/train_pytorch.py#L201-L202

In the above line, the `log` variable is stored in GPU, but for each sample in validation or testing, it is appended to the `logs` variable (a list).

I found that this issue could be fixed by the following code (using function `.cpu()`).

```python
log = model.forward_test_wikikg(query, ans, candidate, sampler.mode, gpu_id).cpu()
```


Thank you for catching this! We have pushed the fix.
Repository: snap-stanford/ogb
Issue: RuntimeError: [enforce fail at CPUAllocator.cpp:67] . DefaultCPUAllocator: can't allocate memory
Root cause: Resource Mishandling

  Hello, when I run the file: ogb-master/examples/lsc/mag240m/gnn.py, this error （RuntimeError: [enforce fail at CPUAllocator.cpp:67] . DefaultCPUAllocator: can't allocate memory）appears when converting the adjacency matrix（gnn.py line 61）. How should I modify the code to run on my device ？
  Or is it because my device cannot complete this work? The configuration of my device is 64G RAM, four 2080ti graphics cards, 11G single-card video memory, and 16T hard disk. 
  If it is the reason for my device, what kind of configuration is required at least to complete this work? 
  Thank you very much! 
  Best wishes!

It seems like the `adj_t.to_symmetric()` call will consume more than 64GB. The resulting adjacency matrix will consume about 39GB, and the intermediate process of generating reverse edges will consume some additional memory, too. I'm not sure though if there exists more memory-efficient implementations for creating undirected graphs. We tested the code with 128GB of memory which is sufficient to run most examples.

Thank you very much for your answers！  I will try to use a server with larger memory.
Repository: snap-stanford/ogb
Issue: Batching Graphs in PygPCQM4MDataset
Root cause: Unimplemented Feature Gap

Hi thanks for preparing the processing code!

Was just thinking if batching the smiles graph into separate torch files would be a feasible solution to reduce memory requirement? I notice in the `process()` function of `class PygPCQM4MDataset(InMemoryDataset):`, the list of graphs obtained from the smiles strings are all combined into a single dataset, and subsequently `torch.save`'d into one file (only to be split again later on to different `dataloader`s? during training and testing)

Since all of the graphs are independent of each other, would it be possible to perhaps save these into a couple of torch files, each made of batches of several graphs `data` to reduce RAM requirement?

Thanks!

wait i think this is more suitable for the discussion forum sorry
Repository: snap-stanford/ogb
Issue: run.sh maybe have some bug?
Root cause: Documentation Deficiency

hi,
I think run.sh (https://github.com/snap-stanford/ogb/blob/master/examples/lsc/wikikg90m/run.sh) has some problem, the follow cmd --no_save_emb represent not save MLP model param, how can load model when use dglke_eval in test data?

dglke_train --model_name TransE_l2 \
--hidden_dim 200 --gamma 10 --lr 0.1 --regularization_coef 1e-9 \
--valid --test -adv --mix_cpu_gpu --num_proc 4 --num_thread 4 \
--gpu 0 1 2 3 \
--async_update --force_sync_interval 10000 --**no_save_emb** \
--print_on_screen --encoder_model_name roberta --save_path $SAVE_PATH

Is this as expected？？

![image](https://user-images.githubusercontent.com/16524285/114299030-d535d880-9aeb-11eb-8aeb-5a989c172ee9.png)


Hi, feel free to remove --no_save_emb to save the model if you want. 

It is not a bug and you do not need to load the model to achieve the test prediction. When you train the model, it will automatically save the valid prediction and test prediction every certain steps. After training, you can directly run `save_test_submission.py` to save the test prediction file according to the best validation performance, check the details at https://github.com/snap-stanford/ogb/tree/master/examples/lsc/wikikg90m#saving-test-submission.
Repository: snap-stanford/ogb
Issue: GPU utilization rate

I run ogb.lsc.ComplEx_roberta example on 4x16G machine. However, the GPU utilization rate is 0% at most time, and the GPU occupation memory is 1300+MB per card. Is it a normal phenomenon?  Thank you!

Hi! It seems to be related to loading roberta feature from the disk: https://github.com/snap-stanford/ogb/issues/131
Did you observe the same issue when you run ComplEx_shallow?

Yes. Actually I met a strange phenomenon. I run ogb.lsc.ComplEx_shallow on another machine(8x16G). After loading the model and print the arguments, all the process seem stopped. All GPU utilization rate is 0% , all GPU occupation memory is 1351MB, and even the CPU% is 0... 

Do you think I should replace entity_feat to all_entity_feat?

Hi, can you check whether the training freezes? Also are you running the example scripts?

Yes, the training freezes. And I'm exactly running the example scripts `run.sh`. And I don't know why...
I have now replaced `dataset.entity_feat` to `dataset.all_entity_feat` and run ComplEx_concat to see the results.

What is the size of the memory of your machine? Have you seen a line "Model created, it takes ... seconds" printed?

Yes, and it freezes after printing the model arguments. This is ComplEx_concat example, and the memory of the machine is 400G+. I don't know why it freezes. No error messages, just hang.

<img width="511" alt="屏幕快照 2021-04-14 下午2 09 02" src="https://user-images.githubusercontent.com/20554008/114662287-005a3b00-9d2b-11eb-9f12-6e8b62668fbd.png">
<img width="273" alt="屏幕快照 2021-04-14 下午2 08 04" src="https://user-images.githubusercontent.com/20554008/114662314-0c45fd00-9d2b-11eb-8408-15ce42d24969.png">


This is really weird, it runs smoothly on my machine. Can you send me an email at hyren@cs.stanford.edu, I can learn more details about your case and help you fix this issue.

Ok.
Repository: snap-stanford/ogb
Issue: Regression task dataset generation
Root cause: Documentation Deficiency

Hi, I am trying to use ogb for regression task. 
[https://github.com/snap-stanford/ogb/blob/68a303f320220cda859e83e3a8660f2b9debedf6/ogb/io/save_dataset.py#L12]
Using DatasetSaver class, I am trying to save my own molecular dataset. However, I feel it is also built for classification tasks. Can you guide on how to store regression dataset after generating graph data and use that further to evaluate? Is it already mentioned in the repo and I am missing it? Can you point towards that? 

Thanks. 



You can save the regression task as well. Below is how you can do it---just modifying sections 3 and 6 in [README](https://github.com/snap-stanford/ogb/blob/master/ogb/io/README.md).

```python
from ogb.io import DatasetSaver
saver = DatasetSaver(dataset_name = dataset_name, is_hetero = False, version = 1)
...
# 3. Saving target labels
labels = np.random.randn(num_data,num_tasks).astype(np.float32) # just save regression values
saver.save_target_labels(labels)

# 6. Saving task information
save_task_info(task_type = 'regression', eval_metric = 'mae') # set task_type as regression, eval_metric is your choice.
```

Hope this helps!


Yes. It worked. Thank you so much for your prompt response.
Repository: snap-stanford/ogb
Issue: gnn training stuck
Root cause: Resource Mishandling

![image](https://user-images.githubusercontent.com/26788062/114641148-25878300-9d04-11eb-9fd2-063f898fb105.png)
when I trained gnn baseline, it got stuck more than 20min after displaying the parameter , is it normal?
Thanks 

Hi! See https://github.com/snap-stanford/ogb/issues/131 for the potential solution. Your hard disk needs to be fast, or you need to put all the features into CPU memory.

it works, Thanks.
One more question, how can I use multi-gpu for training, when I change the trainer setting, like gpus=[0,1,2,3,4,5,6,7].
It will throw a error like:
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
Traceback (most recent call last):
  File "rgnn.py", line 416, in <module>
    trainer.fit(model, datamodule=datamodule)
  File "/home/user/miniconda/envs/py36/envs/python37/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 499, in fit
    self.dispatch()
  File "/home/user/miniconda/envs/py36/envs/python37/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 546, in dispatch
    self.accelerator.start_training(self)
  File "/home/user/miniconda/envs/py36/envs/python37/lib/python3.7/site-packages/pytorch_lightning/accelerators/accelerator.py", line 73, in start_training
    self.training_type_plugin.start_training(trainer)
  File "/home/user/miniconda/envs/py36/envs/python37/lib/python3.7/site-packages/pytorch_lightning/plugins/training_type/ddp_spawn.py", line 108, in start_training
    mp.spawn(self.new_process, **self.mp_spawn_kwargs)
  File "/home/user/miniconda/envs/py36/envs/python37/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/home/user/miniconda/envs/py36/envs/python37/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 179, in start_processes
    process.start()
  File "/home/user/miniconda/envs/py36/envs/python37/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/home/user/miniconda/envs/py36/envs/python37/lib/python3.7/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/home/user/miniconda/envs/py36/envs/python37/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/home/user/miniconda/envs/py36/envs/python37/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/home/user/miniconda/envs/py36/envs/python37/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/home/user/miniconda/envs/py36/envs/python37/lib/python3.7/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
MemoryError


In case you load all node features directly into memory, it might want to make use of shared memory to ensure that different processes do not need to replicate the data.

Thanks