zhiminy/EvalEng huggingface dataset:
Repository: snap-stanford/ogb
Issue: Ratio of positive labels in test set is very different from training or validation set in proteins function dataset
Root cause: Documentation Deficiency

Hi authors,

May I ask why the label distribution of test set is so different? For example label 0:
`split_idx = dataset.get_idx_split()`
`train_nodes = split_idx['train'].numpy()`
`valid_nodes = split_idx['valid'].numpy()`
`test_nodes = split_idx['test'].numpy()`
`print(data.y[train_nodes,0].sum()/len(train_nodes))`
`print(data.y[valid_nodes,0].sum()/len(valid_nodes))`
`print(data.y[test_nodes,0].sum()/len(test_nodes))`
And the results are: `0.6287, 0.6269, 0.0847` respectively. Is this intentional?

Thanks!



Hi!

This is a great realization. This is because of the species split, but I was not expecting such a huge difference in the ratio of positive ratios. To give you more context below is the positive label ratio for each species, which does vary significantly across different species:

```python
from ogb.nodeproppred import PygNodePropPredDataset
import torch
dataset = PygNodePropPredDataset('ogbn-proteins')
data = dataset[0]

unique_species = torch.unique(data.node_species)
for species_id in unique_species:
    print('species id: ', int(species_id))
    node_idx = torch.nonzero(data.node_species == species_id)[:,0]
    print(float(data.y[node_idx].sum()/(len(node_idx)*112))) # 112 tasks
    print()
```

Output
```bash
species id:  3702
0.0976378545165062

species id:  4932
0.1544351875782013

species id:  6239
0.06898193806409836

species id:  7227
0.12101421505212784

species id:  7955 (test species)
0.022200047969818115

species id:  9606
0.21192370355129242

species id:  10090 (validation species)
0.1685332953929901

species id:  511145
0.1294090896844864
```


The good news is that our evaluation metric is ROC-AUC, which should be pretty robust to the difference in positive label ratio.

> The good news is that our evaluation metric is ROC-AUC, which should be pretty robust to the difference in positive label ratio.

I see. Thanks!
Repository: snap-stanford/ogb
Issue: ogbn-mag edge features
Root cause: Documentation Deficiency

Hi there, 
I am looking at the example given regarding ogbn-mag dataset. From the paper it is stated that ogbn-mag dataset has edge-features, but I don't see it being utilised. Can I know where can I get the edge features and what does the edge features represent?

![image](https://user-images.githubusercontent.com/19768905/135586170-32e4fb0a-3ab3-4b02-b8ed-b6bcc363d7c0.png)


Good point. ogbn-mag is a heterogeneous graph. We regard the edge type as edge feature.
Repository: snap-stanford/ogb
Issue: About how to encode Domain index ? 

Hi, OGB Team, for ogbg_proteins and ogbl_ppa, i was wondering how you encode the species index into node features? just append the species index into each node feature or append the one-hot encoding? For each graph, since it only belongs to one species domain, do you encode one species index into all node features within one graph? And for ogbg-ppa, ogbg-molhiv, ogbg-molpcba, do you encode the species index into node or edge feature? Thanks !

For ogbg-ppa (graph classification), we did not use the species index, as it is annotated at the level of the entire graph. For ogbn-proteins (node classification) and ogbl-ppa (link prediction), you can use species id as the node feature.

The ogbg-mol* datasets are all molecule datasets; hence you do not have the notion of species.
Repository: snap-stanford/ogb
Issue: No Positive Samples in MolPCBA Test Set Assay #45
Root cause: Documentation Deficiency

Minimal reproducible example:
```python
from ogb.graphproppred import PygGraphPropPredDataset
from torch_geometric.data import DataLoader
import torch

dataset = PygGraphPropPredDataset(name = 'ogbg-molpcba')
split_idx = dataset.get_idx_split()
print(sum([torch.nan_to_num(point.y) for point in dataset[split_idx['test']]])[0, 45])
```

This results in an output of 0 meaning there are no positive examples for this specific endpoint (index 45). How can we properly calculate the overall average precision in this case? This metric is undefined with zero positive samples.

You are right. The label index 45 does not have any positive test examples, and the Average Precision cannot be defined on the label.
Our evaluator actually skips labels that do not have any positive examples: see [here](https://github.com/snap-stanford/ogb/blob/master/ogb/graphproppred/evaluate.py#L196-L199), so you won't experience the error if you use our evaluator.
Repository: snap-stanford/ogb
Issue: Separate leaderboard by transductive and inductive learning
Root cause: Documentation Deficiency

Hi,

I noticed that GraphSAINT on OGB-products was running under an inductive learning setting, where the training graph only consists of training and validation nodes.  However, ClusterGCN, full-batch GraphSAGE in the example implementations, and DGL's implementation of GraphSAGE all follow a transductive setting where the test nodes are included in the training graph as well (albeit without labels).

I think separating the two scenarios in different leaderboards would be fairer.  What do you think?

Thanks.

Hi, and thanks for this interesting issue. We use an inductive scenario for GraphSAINT because the method is defined to only work in inductive learning settings. Nonetheless, one could also extend GraphSAINT to work in transductive learning scenarios (with modified loss normalization coefficients), but we decided to follow the original implementation as much as we can.

Concerning your request, we decided to not provide different leaderboards for this dataset because we want users to exploit the transductive nature of this task. Nonetheless, If a method does only make use of inductive information and can compete with transductive ones, we are still very happy to include it in our leaderboards. The results of GraphSAINT already suggest that inductive methods may very well be able to compete with transductive ones on `ogbn-products`.
Repository: snap-stanford/ogb
Issue: Cannot import PygPCQM4Mv2Dataset
Root cause: Environment Incompatibility

OGB version 1.3.2, Python 3.8.10.

Simply running
```python
from ogb.lsc import PygPCQM4Mv2Dataset
```
as per the instructions, returns
`ImportError: cannot import name 'PygPCQM4Mv2Dataset' from 'ogb.lsc' (/opt/miniconda3/envs/ogb/lib/python3.8/site-packages/ogb/lsc/__init__.py)`

Edit: the same happens with the standard `PCQM4Mv2Dataset`, too.

Did you install rdkit?

I thought I had, but you're right, it was a different environment! Thanks 😳
Repository: snap-stanford/ogb
Issue: Is original source code for ogbg-code2 available?
Root cause: Documentation Deficiency

I'd like to test existing code summarization techniques other than GNNs (e.g., Allamanis [2016], Alon [2018, 2019], etc.) on the same dataset as ogbg-code2, but OGB package only downloads preprocessed graph data and we can't use it for source code specific methods.
Could you provide a raw source code dataset? 

The original source code is provided under `ogbg_code2/mapping/`.

I found it. Thank you for the quick reply.
Repository: snap-stanford/ogb
Issue: Small error in example in docs

Thanks a lot for making this collection of datasets. I found a small error in the docs.

On https://ogb.stanford.edu/docs/nodeprop/ , in the example code of the DGL loader, the code example is
```
from ogb.nodeproppred import DglNodePropPredDataset

dataset = NodePropPredDataset(name = d_name)
```
but should be
```
from ogb.nodeproppred import DglNodePropPredDataset

dataset = DglNodePropPredDataset(name = d_name)
```


Thanks for catching the typo. This is now fixed!
Repository: snap-stanford/ogb
Issue: Value error 
Root cause: External Dependency Breakage

I am trying to run the code snippet provided by the authors,  `from ogb.graphproppred import PygGraphPropPredDataset` but the following ValueError 

`ValueError: module functions cannot set METH_CLASS or METH_STATIC`

I am trying to run this code on colab

Could be related to this: https://github.com/pandas-dev/pandas/issues/23040
Repository: snap-stanford/ogb
Issue: Unable to import smiles2graph 
Root cause: Environment Incompatibility

I am trying to import `from ogb.lsc.pcqm4m_pyg import PygPCQM4MDataset` but constantly getting this error 

`ImportError: cannot import name 'smiles2graph' from 'ogb.utils' (/usr/local/lib/python3.7/dist-packages/ogb/utils/__init__.py)`.

Did you install `rdkit`?
Repository: snap-stanford/ogb
Issue: ogbn-products training and testing split
Root cause: Documentation Deficiency

Hi, OGB team:

I was wondering if you have the source code for ogbn-products training and testing split, leaderboard says you use the sales ranking (popularity) to split nodes into training/validation/test sets. Where can i get the sales ranking?

Thanks!

We do not provide the sales ranking, but we just used the meta data provided in the original Amazon dataset.

Hello @weihua916,

Do you, by any chance, have the raw text Amazon dataset from which ogbn-products was generated? In particular, I'm interested to map graph node features to text descriptions and labels to product text labels. I'm looking for something similar to the raw text files you provide for ogbn-arxiv and ogbn-papers100M. I did not find it in the dataset linked in Cluster-GCN. I also tried to download Amazon3M from [here]( http://manikvarma.org/downloads/XC/XMLRepository.html), but the mapping is not straightforward.

Any help or guidance will be much appreciated.
Best regards,
Hani

Hi Hani,
@HaniItani 
You can refer https://github.com/amzn/pecos/tree/mainline/examples/giant-xrt for the text datasets.
Repository: snap-stanford/ogb
Issue: biokg bug_fix
Root cause: Documentation Deficiency

**biokg_bug_fix:**

 the run.py file in biokg has a bug when it is running on linux which essentially stops the model from reporting the train/valid/test results gradually at each step which is confusing for users

Below PR is my fix for this issue
https://github.com/Zahra-Bakhtiari/biokg_bug_fix/pull/1/files

Thanks for PR! Why not just pass `--print_on_screen`?

Maybe I did not explain it fully. I trained RotatE and pRotatE on ogbl-biokg model on AWS. It took more than 12 hours and during the first try nothing was showing up on the screen so I had no idea what would be the state of the model and when I can turn off EC2 resources, this fix resolved the issue 

Yes, I understand. If you want something on the screen, you just add the turn on the argument `--print_on_screen`.
Repository: snap-stanford/ogb
Issue: ogbn_arxiv: invalid gradient error if the directed edges don't be converted to undirected
Root cause: Validation Gap

Hi @rusty1s , for **ogbn_arxiv** dataset, I commented out `edge_index = to_undirected(edge_index, data.num_nodes)` and set `args.use_sage` be true, while a runtime error occurred when statement `loss.backward()` was executed. 

> RuntimeError: Function torch::autograd::CppNode<SPMMMean> returned an invalid gradient at index 3 - got [169342, 256] but expected shape compatible with [169343, 256]

Any ideas about this? Thanks!

You may need to set the size of the `SparseTensor`, e.g.:
```python
adj = SparseTensor(row=edge_index[0], col=edge_index[1], sparse_sizes=(data.num_nodes, data.num_nodes))
```

Yep you are right! I just tried and that error disappeared. Thank you very much :)
Repository: snap-stanford/ogb
Issue: ogbg-code2 returns F1=0. all the time
Root cause: External Dependency Breakage

Hi OGB team, first of all, thank you very much for this great resource and your work!

When I run `examples/graphproppred/code2/main_pyg.py`, I get F1=0. all the time, for train, valid, and test set. 

In the first screenshot, I'm showing the results for the first three epochs only. This is what I get by just running your script and not touching anything, not even the logging. 
In the second screenshot, you can see that the training loss doesn't improve much even over the default 25 epochs. Again, the final F1 scores here are all 0.   

![image](https://user-images.githubusercontent.com/11850255/147456371-6fd1ed94-36aa-4b83-88ea-597475ca5772.png)
![image](https://user-images.githubusercontent.com/11850255/147456932-ca1057d3-2610-492b-8056-e6a774b3a407.png)


Hi! This might be related to https://github.com/snap-stanford/ogb/issues/260 Can you reprocess the dataset after you install/update PyG?

Thanks for your response! I just re-installed PyG from master, and ran the script again after deleting the folder `dataset` entirely (so the data was downloaded and processed again). It didn't help. I still get F1 scores either being exactly zero 0 or almost zero. 
![image](https://user-images.githubusercontent.com/11850255/147653713-888f7597-1772-4d15-9756-27b74ca3b21a.png)


Hi! I used Pytorch version `1.10.0` and was able to reproduce the same issue. It turns out that the root cause is Pytorch's bug on CrossEntropyLoss (see [here](https://github.com/pytorch/pytorch/issues/67167))!

The issue has been resolved in [`1.10.1`](https://github.com/pytorch/pytorch/releases/tag/v1.10.1). Once I updated the Pytorch version, I obtained non-zero F1 scores:

[10030 rows x 2 columns]
#Params: 13841815
=====Epoch 1
Training...
Iteration: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3188/3188 [05:33<00:00,  9.56it/s]
Average training loss: 3.1366342785516976
Evaluating...
Iteration: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3188/3188 [03:02<00:00, 17.43it/s]
Iteration: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 179/179 [00:13<00:00, 13.02it/s]
Iteration: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 172/172 [00:11<00:00, 14.94it/s]
{'Train': {'precision': 0.09623143681654477, 'recall': 0.05144160132413116, 'F1': 0.06412746542938594}, 'Validation': {'precision': 0.08581320944909497, 'recall': 0.04399707889759167, 'F1': 0.05549395431281744}, 'Test': {'precision': 0.08448150173136505, 'recall': 0.043179188326809975, 'F1': 0.05463354811359185}}


Thank you so much, upgrading to `1.10.1` indeed solved it!
Repository: snap-stanford/ogb
Issue: "ConnectionResetError: [Error 104] Connection reset by peer" when downloading ogbn-papers100M
Root cause: Environment Incompatibility

Dear Ogbn Team,
  I have tried to downloaded the ogbn-papers100M dataset. But everytime when it reach about 1 to 4%, the download process would be ceased and this error raised.
  
  Could you please help me with this issue?

Best

btw I am using dgl to download the dataset

Hi! I am not encountering the issue. Can you paste the code snippet and the error? Also, make sure you are using a reliable internet connection!
Repository: snap-stanford/ogb
Issue: examples/linkproppred/citation2/cluster_gcn.py crashes during evaluation
Root cause: External Dependency Breakage

Tested with `torch 1.10.1` and `torch-geometric 2.0.3`
```
  File "/home/jean/Research/Projects/ogb/examples/linkproppred/citation2/cluster_gcn.py", line 223, in main
    result = test(model, predictor, data, split_edge, evaluator,
  File "/home/jean/anaconda3/envs/ogb/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
    return func(*args, **kwargs)
  File "/home/jean/Research/Projects/ogb/examples/linkproppred/citation2/cluster_gcn.py", line 119, in test
    weights = [(conv.weight.cpu().detach().numpy(),
  File "/home/jean/Research/Projects/ogb/examples/linkproppred/citation2/cluster_gcn.py", line 119, in <listcomp>
    weights = [(conv.weight.cpu().detach().numpy(),
  File "/home/jean/anaconda3/envs/ogb/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1177, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'GCNConv' object has no attribute 'weight'
```

Should it be `conv.lin.weight.t()` instead?

Hi! Thanks for reporting! @rusty1s Could you please update the example code to be compatible with PyG 2.0?

Should be fixed in the example, see https://github.com/snap-stanford/ogb/pull/295.
Repository: snap-stanford/ogb
Issue: AttributeError: 'GlobalStorage' object has no attribute 'num_nodes'
Root cause: Validation Gap

Hi!
I got an error when run 
**python3 examples/linkproppred/ddi/gnn.py**

Environments
pytorch: `1.10.0`
torch_geometric: `2.0.3`

> Namespace(device=0, log_steps=1, use_sage=False, num_layers=2, hidden_channels=256, dropout=0.5, batch_size=65536, lr=0.005, epochs=200, eval_steps=5, runs=10)
Downloading http://snap.stanford.edu/ogb/data/linkproppred/ddi.zip
Downloaded 0.04 GB: 100%|███████████████████████████████████████████████████████████████████████████████████| 46/46 [00:41<00:00,  1.12it/s]
Extracting dataset/ddi.zip
Processing...
Loading necessary files...
This might take a while.
Processing graphs...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 48.51it/s]
Converting graphs into PyG objects...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 2250.16it/s]
Saving...
Done!
/data/huangxikun/opt/miniconda3/envs/gnn/lib/python3.9/site-packages/torch_geometric/data/storage.py:259: UserWarning: Unable to accurately infer 'num_nodes' from the attribute set '{'edge_index'}'. Please explicitly set 'num_nodes' as an attribute of 'data' to suppress this warning
  warnings.warn(
Traceback (most recent call last):
  File "/data/huangxikun/opt/miniconda3/envs/gnn/lib/python3.9/site-packages/torch_geometric/data/storage.py", line 48, in __getattr__
    return self[key]
  File "/data/huangxikun/opt/miniconda3/envs/gnn/lib/python3.9/site-packages/torch_geometric/data/storage.py", line 68, in __getitem__
    return self._mapping[key]
KeyError: 'num_nodes'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/data/huangxikun/workspace_py/OBG_xk/ogb/examples/linkproppred/ddi/gnn.py", line 296, in <module>
    main()
  File "/data/huangxikun/workspace_py/OBG_xk/ogb/examples/linkproppred/ddi/gnn.py", line 245, in main
    emb = torch.nn.Embedding(data.num_nodes, args.hidden_channels).to(device)
  File "/data/huangxikun/opt/miniconda3/envs/gnn/lib/python3.9/site-packages/torch_geometric/data/data.py", line 362, in __getattr__
    return getattr(self._store, key)
  File "/data/huangxikun/opt/miniconda3/envs/gnn/lib/python3.9/site-packages/torch_geometric/data/storage.py", line 50, in __getattr__
    raise AttributeError(
AttributeError: 'GlobalStorage' object has no attribute 'num_nodes'

It seems that pyg dataloader does not set 'num_nodes' as an attribute of 'data'. I notice that there is a file `dataset/ogbl_ddi/raw/num-node-list.csv.gz` which contains the number of nodes of ddi network. So I think  pyg dataloader should set this attibute, right?
Thanks!

This is a bug in PyG which is now fixed, see https://github.com/pyg-team/pytorch_geometric/commit/af95df956f4473656f94d9ed04be0480534e397c.

Nonetheless, the `DDI` dataset should set the `num_nodes` attribute. This is now fixed inside OGB as well, see https://github.com/snap-stanford/ogb/pull/295.
Repository: snap-stanford/ogb
Issue: Using a custom dataset
Root cause: Documentation Deficiency

How do you prepare a obg dataset from a csv file?

Try our dataset saver: https://github.com/snap-stanford/ogb/blob/master/ogb/io/README.md

In the last section, what do you mean by sending those files to you? How should they be shared

For your use case, you only need to do until section 8. The last section is for people to contribute their datasets

cool, thanks a lot
Repository: snap-stanford/ogb
Issue: some classes are missing in ogbn-products training and validation splits
Root cause: Validation Gap

I just noticed that in the ogbn-products dataset, some classes are present in the test split, but are missing in the training and/or validation splits. More specifically, out of 47 total classes (0 to 46) appearing in the test set, five classes (42 to 46) never appear in the training set, and thirteen classes (34 to 46) are not available in the validation set.

<img width="644" alt="Screen Shot 2022-02-13 at 21 11 32" src="https://user-images.githubusercontent.com/3238345/153767486-98d1b409-1027-4303-82f4-f27be39e0e9b.png">


I am wondering if this is intentional?

Hi, thanks for finding this out. This is not intentional.

Basically, around 1.6% of test data points are assigned with labels never seen during the training stage; hence, we cannot expect any models to perform well on those data points.
Repository: snap-stanford/ogb
Issue: [Remark] Dataset splitting section of ogbn-arxiv

Hello OGB community,

Really nice effort of standardizing the ML on Graphs with a benchmark!

As I read and play around with OGB I noted that in the **Dataset Splitting** section for **[oghn-arxiv](https://ogb.stanford.edu/docs/nodeprop/#ogbn-arxiv)** you essentially describe again the splitting used for the **ogbn-products** dataset.

You might want to update it. :)

Best,
Makis

Hi Makis, thanks for catching the typo. I just fixed it!
Repository: snap-stanford/ogb
Issue: [ogbn-product] labelidx2productcategory.csv: missing and strange product categery names
Root cause: Documentation Deficiency

Hi there,

It seems like the following product category names are missing (or strange), especially 24, 45, and 46. I was wondering what are the correct texts for these category names? Do you have an updated version of this csv file?

![image](https://user-images.githubusercontent.com/36200682/154915276-53e00534-9694-4e40-b0e9-ac315be63cff.png)
![image](https://user-images.githubusercontent.com/36200682/154915405-d7144952-9dff-4738-a3ef-527cfaf7196d.png)

Thanks.

Hi! Good point. The dataset is originally curated by the [ClusterGCN paper](https://arxiv.org/pdf/1905.07953.pdf), so it is hard to say what exactly those labels are.
Repository: snap-stanford/ogb
Issue: test-dev evaluation for WikiKG90Mv2 dataset
Root cause: Others

Hi

I had submitted test-dev output for my model for WikiKG90Mv2 dataset using the following form: https://ogb-save.stanford.edu/leaderboard/

Can you please confirm if you have received the submission? The email address used is apoorvumang@gmail.com . It has been almost 1 week and I did not receive any confirmation email of my submission, hence I am asking.

Thank you
Apoorv

@weihua916 @hyren can you please help with this? Thanks