zhiminy/EvalEng huggingface dataset:
Repository: snap-stanford/ogb
Issue: Workflow for contributing additional graph resources?
Root cause: Documentation Deficiency

Hi OGB team,

We recently released a large-scale (approx. 5 million edges) benchmark for biomedical link prediction called OpenBioLink (https://github.com/OpenBioLink/OpenBioLink). Would you be interested in adding such a dataset to OGB, and what would be the suggested workflow for making this happen?

Apologies for the delayed response. Thanks for the reference, and they are definitely interesting. For now, we are focusing on our first release of OGB; after that, we can discuss whether to include your dataset in our OGB datasets. Thanks!

Thanks a lot for the suggestion. The dataset is very interesting. However, we already have multiple biomedical link prediction datasets, and we are planning to release more of them in the next release. Therefore, we are not planning to include the dataset in OGB. 

Dear @weihua916 

Thanks for the response. Please note that in the meanwhile, the peer-reviewed paper on the OpenBioLink benchmark has appeared in 'Bioinformatics' https://academic.oup.com/bioinformatics/advance-article/doi/10.1093/bioinformatics/btaa274/5825726

We are working on establishing OpenBioLink as the premier benchmark for link prediction with general-purpose biomedical knowledge, so please let us know if we can contribute some resources (including contributing code) to align OpenBioLink with OGB! :)

Congratulations!! That's pretty cool.

We will think more about how we may collaborate. We will let you know if we want your contributions. Thanks!

Hi OGB team,

We would be very glad if you would accept our contribution. Since it appears that you don't have a heterogenous graph for link prediction, we think OpenBioLink would be a great addition. OpenBioLink covers a big range of entities in the biomedical field (Anatomy, Gene/Protein, Diseases, Drugs, Pathways, ...) and would even provide negative edges (from biomedical databases and randomly generated).

I am working right now on a format of the OpenBioLink dataset that would be compatible with OGB. Please let me know if you want to include it.

Hi! Thank you very much for your interest and detailed explanation. 

Many heterogeneous graph datasets will be included in our next release (including biomedical knowledge graph that we have curated ourselves). We will figure out how your dataset fits in OGB after our next release. Thanks!

Hi! Apologies for the delay. I would like to tell you that we have finally set up the official workflow of dataset contribution: [website](https://ogb.stanford.edu/docs/dataset_overview/) and [doc](https://docs.google.com/document/d/1bSVtHGtSMwu6B3setIrEAnNxI4qBYRoFIClAkk7aOAg/edit?usp=sharing). Hope the guideline helps!

Thanks!
Repository: snap-stanford/ogb
Issue: Torch Requirement for Library-Agnostic Modules
Root cause: Architectural Constraint

Hi, I'm a TensorFlow user and thanks very much for providing library-agnostic modules. 

There are some *.pt format data in the raw data, which is hard to handle without torch. Although the pull request #102 can successfully load them with some pickle code, I don't think they can still work if ogb put more complex torch-format data in the raw dataset. So I suggest ogb to use pickle instead of torch-format data (such as *.pt) in the raw data for furture datasets. 

Thank you in advance!





Thanks for your suggestion.

Isn't it easier to just install `torch`, even if that's only used for loading the raw data?

Thanks for your answer. Although installing torch is easy, I think most users who use TensorFlow or MXNet are geeks who do not want to use their favorite deep learning libraries with torch at all. It's just like many torch users not like TensorFlow/MXNet. For example, do you think a TensorFlow/MXNet GNN library will provide demos that require users to install torch? I think the authors may provide a wrapper inside their libraries using ogb dataset urls instead.

In tems of this issue, I also do not suggest ogb to substitute the existing datasets since they are already downloaded by a lot of users. I'm just worried about if my pull request can support the new datasets you introduced in the furture.

Thanks for your again!



Thanks for your suggestion. I understand it, but I find `torch.load` and `torch.save` to be more well-behaved than `pickle` especially when handling large file size. I'd like to keep the package as it is, also to prevent potential bugs that we may introduce by merging the new pull request. Thank you for your consideration, and I hope you understand this.

Thanks very much, I really understand it.
Repository: snap-stanford/ogb
Issue: Question about ogbn-proteins dataset

Hi, 
According to the description of ogbn-proteins dataset, the 112 kinds of labels correspond to Gene Ontology (GO) functions. Why, among all the functions defined in the GO, these 112 labels were selected?
Thanks in advance! 

Hi! This is a great question. We consider GO functions that are fine-grained (leaves of the GO hierarchy) and have enough positive labels to train models on. That's how we have arrived at the 112 kinds of labels.

Thanks for your prompt response!  
However, GO: 0003674, a root of the GO hierarchy, isn't it one of the kinds of labels? 
I also take the opportunity to ask what was the number of positive labels that you considered as enough.


You are welcome! I double-checked my code, and you are right. Let me correct my previous response. 
The following is what we did:

We first get a list of GOs that are **immediate parents of** the leave GOs.
Out of those GOs, we extracted GOs that appear more than 200 times in each of train/valid/test splits. This gives us the 112 GOs in our dataset.
Repository: snap-stanford/ogb
Issue: ogb-linkproppred-evaluate-hits@k
Root cause: Documentation Deficiency

Hello, 

I read your code about hits@k. 
```
def _eval_hits(self, y_pred_pos, y_pred_neg, type_info):

        '''
            compute Hits@K
            For each positive target node, the negative target nodes are the same.

            y_pred_neg is an array.
            rank y_pred_pos[i] against y_pred_neg for each i
        '''

        if len(y_pred_neg) < self.K:
            return {'hits@{}'.format(self.K): 1.}

        if type_info == 'torch':
            kth_score_in_negative_edges = torch.topk(y_pred_neg, self.K)[0][-1]
            hitsK = float(torch.sum(y_pred_pos > kth_score_in_negative_edges).cpu()) / len(y_pred_pos)

        # type_info is numpy
        else:
            kth_score_in_negative_edges = np.sort(y_pred_neg)[-self.K]
            hitsK = float(np.sum(y_pred_pos > kth_score_in_negative_edges)) / len(y_pred_pos)

        return {'hits@{}'.format(self.K): hitsK}
```
I am not sure whether it is the Hits@K that I understand is wrong. Your implementation is: if the K-th largest pred in the negative sample is npk (npk = Top-k(neg_pred)[k]), then: 

<a href="https://www.codecogs.com/eqnedit.php?latex=Hits@K&space;=&space;\textbf{sum}(pos_{pred}&space;>&space;npk)&space;/&space;\textbf{len}(pos_{pred})" target="_blank"><img src="https://latex.codecogs.com/gif.latex?Hits@K&space;=&space;\textbf{sum}(pos_{pred}&space;>&space;npk)&space;/&space;\textbf{len}(pos_{pred})" title="Hits@K = \textbf{sum}(pos_{pred} > npk) / \textbf{len}(pos_{pred})" /></a> 
(That is, the proportion of larger positive samples (> npk) in all positive samples. )

I understand that Hits@K is the proportion of positive samples among the top k values in neg_pred+pos_pred,
<a href="https://www.codecogs.com/eqnedit.php?latex=Hits@K&space;=&space;\textbf{len}(&space;[\textbf{Top-k}(neg_{pred}&space;&plus;&space;pos_{pred})&space;\in&space;\textbf{Pos}]&space;)&space;/&space;K." target="_blank"><img src="https://latex.codecogs.com/gif.latex?Hits@K&space;=&space;\textbf{len}(&space;[\textbf{Top-k}(neg_{pred}&space;&plus;&space;pos_{pred})&space;\in&space;\textbf{Pos}]&space;)&space;/&space;K." title="Hits@K = \textbf{len}( [\textbf{Top-k}(neg_{pred} + pos_{pred}) \in \textbf{Pos}] ) / K." /></a>

which is the real Hits@K, and in the link prediction task, which of the above evaluation is better?
The formula is not well written, I hope it can be understood. 


We follow the standard definition of Hits@K calculation: see e.g., http://nlpprogress.com/english/relation_prediction.html

For each positive edge, we see whether it can be ranked among the top K of the negative edges. Hit@K is the ratio of the positive edges that are ranked successfully by the model.
Repository: snap-stanford/ogb
Issue: EOFError: EOF when reading a line
Root cause: Environment Incompatibility

When I run the model on ogbl-citation dataset using GPU clustering, the program asks the following questions when creating PygLinkPropPredDataset:

> Will you update the dataset now? (y/N)
> This will download 2.14GB. Will you proceed? (y/N)

Then ../ogb/linkproppred/dataset_pyg.py produces the following error:

> Traceback (most recent call last):
> ...
> File "src/dev_ogbl_citation.py", line 149, in _create_dataset
>     root="../data")
>   File "/home/ehwang/miniconda3/envs/ogb/lib/python3.7/site-packages/ogb/linkproppred/dataset_pyg.py", line 54, in __init__
>     if input('Will you update the dataset now? (y/N)\n').lower() == 'y':
> EOFError: EOF when reading a line

It is okay when I manually run the model and put "y" for each question. However, this error happens when I run multiple jobs using GPU at the same time. I would appreciate it if anyone can look into this issue. 

Hi! I am not sure if I understand your issue. Downloading and preprocessing only need to be done once. Once this is finished you can run multiple jobs.

Also, did you upgrade to 1.2.4? `ogbl-citation` has been deprecated. The new dataset is `ogbl-citation2`

Upgrading ogb package to 1.2.4 resolved the issue. Thank you!
Repository: snap-stanford/ogb
Issue: [SSL: CERTIFICATE_VERIFY_FAILED]
Root cause: External Dependency Breakage

When running
```
import torch_geometric.transforms as T
from ogb.nodeproppred import PygNodePropPredDataset

dataset_name = 'ogbn-arxiv'
# Load the dataset and transform it to sparse tensor
dataset = PygNodePropPredDataset(name=dataset_name,
                                 transform=T.ToSparseTensor())
print('The {} dataset has {} graph'.format(dataset_name, len(dataset)))
```
I received the following question:
```
ogbn-arxiv has been updated.
Will you update the dataset now? (y/N)
```
then i sent y and received the following error:
```
SSLCertVerificationError                  Traceback (most recent call last)
/usr/lib/python3.7/urllib/request.py in do_open(self, http_class, req, **http_conn_args)
   1349                 h.request(req.get_method(), req.selector, req.data, headers,
-> 1350                           encode_chunked=req.has_header('Transfer-encoding'))
   1351             except OSError as err: # timeout error

21 frames
SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1091)

During handling of the above exception, another exception occurred:

URLError                                  Traceback (most recent call last)
/usr/lib/python3.7/urllib/request.py in do_open(self, http_class, req, **http_conn_args)
   1350                           encode_chunked=req.has_header('Transfer-encoding'))
   1351             except OSError as err: # timeout error
-> 1352                 raise URLError(err)
   1353             r = h.getresponse()
   1354         except:

URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1091)>
```

Same issue on MacOS Big Sur M1.

Hi @weihua916 ,
I tried other datasets and got the same error.

Same here

This indeed seems to be an issue. It is related our server. We will check and get back to you.

I have just released `ogb=1.2.6`. Downloading should work now. Please update your package, and let me know if it works. Thanks!

it works now thx!

Awesome! Closing the issue.

It works, thank you
Repository: snap-stanford/ogb
Issue: Why ogbg-molhiv has disconnected graphs?

May I ask the question why ogbg-molhiv has disconnected graphs? I thought these graphs are molecules and should naturally be connected? For example, I plotted the graph with index 3732, and it looks like the following. May I ask why node 27, which is a single atom, considered part of this molecule?

![Screen Shot 2021-03-02 at 9 18 31 PM](https://user-images.githubusercontent.com/21331572/109756455-d98dde00-7b9c-11eb-99fb-1d59c6a91be7.png)


Good question. On a very rare occasion, the dataset does contain isolated atoms. Unfortunately, I am not an expert in chemistry, so I do not know the answer, but I am pretty sure the dataset is correct as it is.

Hi! I scrutinized the data a bit and found the original molecule is indeed disconnected, as shown below.


![image](https://user-images.githubusercontent.com/16249234/109844740-463ac400-7c01-11eb-922c-d93764d645a9.png)

Below is ipython notebook script that I used to obtain the figure above.
```python
import pandas as pd
from rdkit import Chem
from rdkit.Chem.Draw import IPythonConsole #Needed to show molecules

molhiv_df = pd.read_csv('ogbg_molhiv/mapping/mol.csv.gz')
smiles = molhiv_df.smiles[3732]
# CCC(O)(C(=O)O)c1cc2n(c(=O)c1CO)Cc1cc3ccccc3nc1-2.[NaH]
mol = Chem.MolFromSmiles(smiles)
mol
```

Hope this helps!

Thank you for the quick response! Yes, it is helpful. I guess I will consult some experts in chemistry to learn more about it. 

Answering for the record: those deattached graphs are so-called salts (e.g. NaCl, KCl), that are not needed in most of the cases. Easiest approach to remove them - keep the largest graph, and remove the smallest.
Repository: snap-stanford/ogb
Issue: "embeddings.pt" not created/saved after running "node2vec.py"
Root cause: External Dependency Breakage

Dear Matthias,

I am having a problem when running your program, altough the ogbn_proteins dataset is downloaded, I receive an error(bellow) right at the end and the embeddings.pt file doesn't seem to be created. 
I am using Ubunto, Pytorch 1.7.1, and the corresponding PyG version.
Do you have any thoughts? Thank you very much for your time!

~$ python3 node2vec.py
/home/asouza/anaconda3/lib/python3.6/site-packages/numba/decorators.py:146: RuntimeWarning: Caching is not available when the 'parallel' target is in use. Caching is now being disabled to allow execution to continue.
  warnings.warn(msg, RuntimeWarning)
Downloading http://snap.stanford.edu/ogb/data/nodeproppred/proteins.zip
Downloaded 0.21 GB: 100%|█████████████████████████████████████████████████████████████| 216/216 [00:48<00:00,  4.45it/s]
Extracting dataset/proteins.zip
Processing...
Loading necessary files...
This might take a while.
Processing graphs...
100%|█████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:03<00:00,  3.48s/it]
Converting graphs into PyG objects...
100%|███████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 7169.75it/s]
Saving...
Done!
Traceback (most recent call last):
  File "node2vec.py", line 58, in <module>
    main()
  File "node2vec.py", line 38, in main
    optimizer = torch.optim.SparseAdam(model.parameters(), lr=args.lr)
  File "/home/asouza/anaconda3/lib/python3.6/site-packages/torch/optim/sparse_adam.py", line 49, in __init__
    super(SparseAdam, self).__init__(params, defaults)
  File "/home/asouza/anaconda3/lib/python3.6/site-packages/torch/optim/optimizer.py", line 47, in __init__
    raise ValueError("optimizer got an empty parameter list")
ValueError: optimizer got an empty parameter list

Should be fixed in master. This is caused by a bug introduced in PyTorch 1.7.0, where
```python
optimizer = torch.optim.SparseAdam(model.parameters(), lr=args.lr)
```
needs to be replaced with
```python
optimizer = torch.optim.SparseAdam(list(model.parameters()), lr=args.lr)
```

Thank you so much!
Repository: snap-stanford/ogb
Issue: No information on how BioKG is constructed
Root cause: Documentation Deficiency

neither the code in this repo nor the associated paper explain where the data comes from in the OGB-BioKG. Can you please point me towards the scripts used to construct this KG?

Hi! Thanks for your interest! 

Marinka (@marinkaz) has created the dataset. She could explain a bit more about how the dataset was created. Thanks!

For a bit more context, we've wrapped the OGB heterogeneous/directed graph loaders as benchmark datasets for PyKEEN (https://github.com/pykeen/pykeen/ - for either using existing KGE models or developing your own) and we would like to improve the documentation about the provenance of these datasets.

Any updates on this?

Unfortunately, I cannot handle this. Could you please reach out to Marinka directly to get the information?

I will close it for now. Please have direct correspondence with Marinka.

Is there a particular way I should reach out to @marinkaz that isn't via a GitHub issue? Maybe it would be good to write up some contributing guidelines and update the README in this repository to better inform potential users if you don't think this is an appropriate forum for giving feedback

Here it is: https://dbmi.hms.harvard.edu/people/marinka-zitnik

Hi there! Is there any official update on this that can be made public? Would appreciate it a lot!

Hello OGB team, 

First of all, thank you for the work in collecting and publishing benchmarks for graph ML. 

I also have the same question as the original post. Are there any updates on this? 
Correct me if I'm wrong but given the characterization of this dataset as "Benchmark", I think there is a need to address this. 

Thank you in advance!
Repository: snap-stanford/ogb
Issue: Is node2vec trained using edges which include the test edges in MLP?
Root cause: Documentation Deficiency

In linkPrediction example like ppa, the embeddings of nodes are firstly learned by node2vec and then used to predict links by MLP. Is the node2vec trained using all edges? If yes, this would contaminate the test data which is predicted by MLP (data leakage issue).

I have checked the edges of test data used by MLP model. It shows that the edges in test data are not included in the training data used by node2vec. Can you please confirm this?
Repository: snap-stanford/ogb
Issue: ImportError: cannot import name 'logger'" when running mlp.py
Root cause: Documentation Deficiency

Hello,

I get the import error bellow when I try to run mlp.py, even though I have logger installed (logger from util-linux 2.31.1). I've tried to pip uninstall logger, and then to install it again. I've tried to substitute the "from logger import Logger" statement to just "import logger", "import logging" or "from logging import Logger" and none of them work (they give off different errors).
I searched a lot about this error but nothing seems to work. Do you have any thoughts?

~$ python3 mlp.py --use_node_embedding
/home/asouza/anaconda3/lib/python3.6/site-packages/numba/decorators.py:146: RuntimeWarning: Caching is not available when the 'parallel' target is in use. Caching is now being disabled to allow execution to continue.
  warnings.warn(msg, RuntimeWarning)
Traceback (most recent call last):
  File "mlp.py", line 9, in <module>
    from logger import Logger
ImportError: cannot import name 'Logger'

Thank you very much for your attention.

The `Logger` is provided directly by OGB, see, e.g., [here](https://github.com/snap-stanford/ogb/blob/master/examples/nodeproppred/arxiv/logger.py). Importing works fine for me. If you have trouble to import, you may want to simply copy-paste the `logger` code directly into the example.

Thank you a lot!
Repository: snap-stanford/ogb
Issue: ImportError: cannot import name 'PCQM4MDataset' from 'ogb.lsc'
Root cause: Environment Incompatibility

Error occurs while importing  PCQM4MDataset.

```
>>> from ogb.lsc import PCQM4MDataset
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'PCQM4MDataset' from 'ogb.lsc' (/home/user/miniconda3/envs/OGB/lib/python3.8/site-packages/ogb/lsc/__init__.py)
```

You need to install [`rdkit`](https://www.rdkit.org/docs/Install.html) in order to use `PCQM4M`.

> You need to install [`rdkit`](https://www.rdkit.org/docs/Install.html) in order to use `PCQM4M`.

Thank you. Now it works!
Repository: snap-stanford/ogb
Issue: On What constitutes external data in PCQM4M-LSC

In [Use of External Data from rules](https://ogb.stanford.edu/kddcup2021/rules/) : "For each dataset, models need to be developed only using the provided data."
In [Overview in PCQM4M](https://ogb.stanford.edu/kddcup2021/pcqm4m/) : "Moreover, predicting the quantum chemical property only from 2D molecular graphs without their 3D equilibrium structures is also practically favorable. This is because obtaining 3D equilibrium structures requires DFT-based geometry optimization, which is expensive on its own."

So, is it allowed to use 3D coordinates in model developing? For example, use **MolToXYZ** from rdkit after this line.
https://github.com/snap-stanford/ogb/blob/c118ba18f0ca13cc8fdf03c6ead13f64316fbe0b/ogb/utils/mol.py#L13
Moreover, is it allowed to use some commonly used package other than rdkit in computational chemistry to get input features, such as **pyscf**?

And other rdkit features: Gasteiger charge, Crippen and others.

Thank you for raising the issue. We will discuss this internally and get back to you!

Hi! We have released [a new set of rules](https://ogb.stanford.edu/kddcup2021/rules/#pcqm4m_time) for the PCQM4M-LSC dataset, which should address your questions here. In short, you can use those packages as long as the test-time-inference can be performed within the limited computational budget. The 3D information can also be used as long as it can be obtained within the budget.

Hope this helps, and please let us know if you have any questions or clarifications!

A misclaim above: For a Mol just generated via MolFromSmiles, to use MoltoXYZBlock to get coordinates, one should do EmbedMultipleConfs, MMFFOptimizeMoluecule, CalcEnergy etc. to get conformation before it.

> For example, use **MolToXYZ** from rdkit after this line.



> Hi! We have released [a new set of rules](https://ogb.stanford.edu/kddcup2021/rules/#pcqm4m_time) for the PCQM4M-LSC dataset, which should address your questions here. In short, you can use those packages as long as the test-time-inference can be performed within the limited computational budget. The 3D information can also be used as long as it can be obtained within the budget.
> 
> Hope this helps, and please let us know if you have any questions or clarifications!

Thanks for the clarifications! Must say the 0.1 sec per molecule budget is well placed lol.
Repository: snap-stanford/ogb
Issue: LSC training example crashes on CPU only machine
Root cause: Validation Gap

I had to modify the code to run properly on CPU, as calls to the PyTorch Lightning Trainers assume GPU availability. See for example line 228 in mag240m/gnn.py, added the accelerator param to make it run.

trainer = Trainer(accelerator ='ddp_cpu', max_epochs=args.epochs,
                  callbacks=[checkpoint_callback],
                  default_root_dir=f'logs/{args.model}')


Thanks for the information! I'd assume most users will have the GPU environment, so I will close this issue for now. 
This is good to have on the [Github discussion page](https://github.com/snap-stanford/ogb/discussions/categories/mag240m-lsc).