zhiminy/EvalEng huggingface dataset:
Repository: snap-stanford/ogb
Issue: Running error with sampler.py in citation2
Root cause: Validation Gap

Hi, I am trying to run the code of sampler.py for the citation2 dataset. If GPU was enabled, it would end up with a runtime error "RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)`". If CPU was used, it would raise an index out of bounds error " File "sampler.py", line 144, in train
    h = model(x[n_id], adjs)
IndexError: index 10139841 is out of bounds for dimension 0 with size 2927963
"
Any help would be appreciated!

Thanks for reporting. I fixed the example script.
Repository: snap-stanford/ogb
Issue: Is there any Label Mapping from integers to real product categories provided in obgn-products dataset ?
Root cause: Documentation Deficiency

I am using obgn-products dataset. To perform the EDA on the dataset I need the names for the categories of product. I am not able to find the mapping of integer labels to the real product category in the mentioned references.

So Could you please help me over here ?

Thanks! 

Never mind there is a separate file inside the dataset.
Repository: snap-stanford/ogb
Issue: mapping between obgn_arxiv and titles in the tsv files
Root cause: Documentation Deficiency

Hi, 

I am trying to figure out how the obgn_arxiv nodes and the titles in the tsv file are related

The titleabs.tsv file has 179720 lines and the graph has 169343 nodes (graph.x.shape == [169343, 128])

I got that the fields of the tsv files are Mag id, titles, abstract but I could not find the mag id in the dataset object.

I loaded the dataset with :
```
from ogb.nodeproppred import PygNodePropPredDataset 
d_name = "ogbn-arxiv" 
dataset = PygNodePropPredDataset(name = d_name)  
graph = dataset[0] # pyg graph object
```
Am I missing something there?

Ok just find the mapping in the downloaded folder which contains the needed information. 
Closing the issue.

Hi, I am trying to download the mapping between the nodeID and paperID, but i am unable to find it under this link: https://snap.stanford.edu/ogb/data/misc/ogbn_arxiv/
I am particularly searching for this file : nodeidx2paperid.csv.gz
Repository: snap-stanford/ogb
Issue: How are text feature representations chosen?

Hi, thank you for your contribution with these datasets. I was wondering - for the graphs that have text features on the nodes (e.g., ogbn-products, ogbn-arxiv), how are the text representation methods chosen? They seem to differ by dataset, e.g., bag-of-words + PCA for ogbn-producs versus average word2vec for ogbn-arxiv, but I didn't see any justification for these choices. 

Tara

Good question! word2vec averaging is used for most of the datasets (ogbn-arxiv, ogbn-papers100M, ogbn-mag, ogbl-collab, ogbl-citation2). For ogbn-products, we used the PCA feature since we directly adopted the dataset from [1], where we only changed the dataset splitting.

[1] Wei-Lin Chiang, Xuanqing Liu, Si Si, Yang Li, Samy Bengio, and Cho-Jui Hsieh. Cluster-GCN: An efficient algorithm for training deep and large graph convolutional networks. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), pp. 257–266, 2019.



Thank you for your response. As a follow-up, are leaderboard submissions with text feature representations not provided by the benchmark allowed? For example, bag of words rather than word2vec for ogbn-arxiv. 

It is allowed. You may explicitly mention that in the method title to make that clear.

FYI: you can get raw text of ogbn-arxiv [here](https://github.com/snap-stanford/ogb/tree/master/examples/nodeproppred/arxiv). For the other datasets, you will need to retrieve text information yourself by using the mapping information stored in `mapping/` of the dataset folder.

Thank you very much!!
Repository: snap-stanford/ogb
Issue: Automated code review for Open Graph Benchmark

Hi!
I am a member of the team developing monocodus — a service that performs automatic code review of GitHub pull requests to help organizations ensure a high quality of code.
We’ve developed some useful features to the moment, and now we’re looking for early users and feedback to find out what we should improve and which features the community needs the most.

We ran monocodus on a pre-created fork of your repo on GitHub https://github.com/monocodus-demonstrations/ogb/pulls, and it found some potential issues. I hope that this information will be useful to you and would be happy to receive any feedback here or on my email pavel@monocodus.com.

If you want to try our service, feel free to follow the link: https://www.monocodus.com
The service is entirely free of charge for open source projects. Hope you’ ll like it :)
Repository: snap-stanford/ogb
Issue: Probably wrong argument in the type checking condition
Root cause: Validation Gap

https://github.com/snap-stanford/ogb/blob/3accaa77c98af743ce8210d55256946702b8da42/ogb/graphproppred/evaluate.py#L51

You check the same argument (`y_true`) twice. Is it intended to be this way?

Good catch. Fixed!
Repository: snap-stanford/ogb
Issue: About noise labels

Hello, thanks for the code and datasets.

Could you tell me whether there possibly are some noise labels in the graph property prediction dataset such as ogbg-molhiv and ogbg-molpcba?

Thanks. ^.^



All the labels are considered gold-standard, but potentially have inherent noise due to measurement error.

> All the labels are considered gold-standard, but potentially have inherent noise due to measurement error.

Thanks. 
I evaluate my model in hiv dataset, and I set a fixed seed. But the results of 10-times experiment (same hyperparameters) have large variance.So, I doubt that there are some noisy lable, maybe.  I try to use some noisy detection methods to solve this, but it doesn't work. Maybe my model or code has some problem.

 Besides this, the result(avg value is about 80) in test dataset is higher than that in val dataset(avg value is about 79), I think it is abnormal. But I don't know why. 

I see. One critical component might be the scaffold split we used. The distributions of train/val/test are all different --- we are testing the out-of-distribution generalization of your model.
Repository: snap-stanford/ogb
Issue: Wrong number of parameters in ogbn-arxiv GCN submission?
Root cause: Validation Gap

The GCN submission for ogbn-arxiv reports 142,888 parameters on the leader-board.  Looking at the code and default settings, it seems to use the following architecture:

- GCNConv(128, 256) (parms = 32,768 (W) + 256 (b) = 33,024) 
- BatchNorm(256) (parms = 256*2 = 512)
- GCNConv(256, 256) (parms = 65,536 (W) + 256 (b) = 65,792) 
- BatchNorm(256) (parms = 256*2 = 512)
- GCNConv(256, 40) (parms = 10,240 (W) + 40 (b) = 10,280) 

This architecture gives a total number of parms of 110,120, which is 32,768 fewer than that reported on the leader-board.  It's probably not a coincidence that this number is equal to the input*hidden dimensions.  Is there some other projection layer coming into play, is the leader-board mistaken, or something else?

Hi! You are right, that was a mistake. When I run `sum(p.numel() for p in model.parameters())` I indeed got 110,120. I have updated the parameter count.
Repository: snap-stanford/ogb
Issue: About the training score of ogbl-ppa

Hi OGB team,
I wonder why the negative edge of the validation set is used instead of random negative sampling in the training set evaluation in the example code. 

Hi,
reporting training performance in link prediction tasks is generally difficult when considering that we do not have access to test edges during training. While I agree that one could sample negative edges that due not occur in the positive training/validation/test set, we opted for this approach due to pure simplicity. Overall, the training performance should just indicate on how well we overfit/underfit the training data, and this should be given in both scenarios.
Repository: snap-stanford/ogb
Issue: How to get raw text (e.g., title or abstract) for the ogbn-papers100M dataset?
Root cause: Documentation Deficiency

I notice that the raw text such as title and abstract is available for the smaller ogbn-arxiv dataset.
However,  I did not found the raw text for the larger ogbn-papers100M dataset.

Is there an easy way that I can download or access to the mapping between paper id and its title/abstract for the ogbn-papers100M dataset?

Hi! Thanks for the inquiry. We are checking if this is possible. Thanks for your patience.

This is now available [here](https://github.com/snap-stanford/ogb/tree/master/examples/nodeproppred/papers100M).
Repository: snap-stanford/ogb
Issue: Efficiently creating undirected graphs in an ogb compatible format
Root cause: Documentation Deficiency

Hello ogb team!

I have created my own graphs in PyTorch Geometric and would like to convert these to PygLinkPropPredDataset in order to use existing algorithms (based on OGB Datasets) for link prediction tasks.

Creating graphs according to the manual has been fairly straightforward, but I am unsure how to deal with the (un-)directed attribute. If I load the graph as a directed graph (from PyTorch Geometric) and set the` dataset_dict[name]['add_inverse_edge'] = True` in the make_master_file.py , I get the  "RuntimeError: add_inverse_edge is depreciated in read_binary" when loading the dataset.

Alternatively, I could load the PyTorch Geometric Dataset as an undirected graph and store all edges (bidirectionally), setting the `dataset_dict[name]['add_inverse_edge'] = False.` However, this would lead to an increase in dataset size, and I am unsure if the 
PygLinkPropPredDataset will understand that the graph is undirected?

What option(s) can you recommend?

Thank you very much for your help!

Hi! 

We suggest one of the followings:
(1) Directly save the undirected graph (with bidirectional edges).
(2) Save directed graph and manually apply [`to_undirected`](https://pytorch-geometric.readthedocs.io/en/latest/_modules/torch_geometric/transforms/to_undirected.html) yourself.

For both above, always use `dataset_dict[name]['add_inverse_edge'] = False`.

Both options will be pretty efficient and are nearly the best you can do. (1) is a bit faster, (2) is more disk-efficient.
Repository: snap-stanford/ogb
Issue: Is 'to_symmetric' on Arxiv dataset appropriate?
Root cause: Documentation Deficiency

In this code(gnn.py), you used 'to_symmtric' to make the adjacency matrix of the Arxiv dataset undirected, but I have two questions about it.

First of all, the Arxiv dataset is a directed graph, according to citation relationship of papers. So, it has published year information of those papers.
But here, if we undirect the adjacency matrix to use GraphSAGE (or other GNN models), doesn't the citation relationship change?

Secondly, in the case of GraphSAGE or GAT with multiple layers, is there no chance to Valid and Test nodes be used as message passing? If so, isn't it a kind of cheating even if the label of the nodes are not shown?

It is just for simplicity. See https://github.com/snap-stanford/ogb/issues/58
Repository: snap-stanford/ogb
Issue: Creating Link Prediction Datasets with multiple graphs
Root cause: Unimplemented Feature Gap

Hi OGB Team,

I am currently working on a link prediction project where I have multiple graphs. 

My initial idea was to generate a link prediction dataset that features multiple graphs (of similar topology) and for instance, train on one graph, test on another graph and validate on a third graph. 

If I generate the ` LinkPropPredDataset` and load multiple graphs into my graph list, I am kindly reminded that ` LinkPropPredDataset` does not support multiple graphs for Link Prediction tasks. 
Are you planning to support this in future releases or is the idea of train/test/valid on different graphs not particularly constructive?
My current circumvention for this would be to load multiple OGB datasets (one for each graph) and pass each to a separate Dataloader in the link prediction step, but I am curious why `LinkPropPredDataset` are currently limited to only one graph.
Thanks for the help!


Hi! Good question! We are not planning to support a link prediction dataset object for multiple graphs. If you would like to use multiple graphs, I would suggest you to use `GraphPropPredDataset` and define your link prediction task on top of it.
Repository: snap-stanford/ogb
Issue: Question concerning PygLinkPropPredDataset - get_edge_split()

Hello OGB Team,

I have looked at several examples from the LinkPropPred Datasets and have a question concerning the train/test/val split.

Let's take an example:

```
dataset = PygLinkPropPredDataset(name = "ogbl-ppa")
split = dataset.get_edge_split()
```

If I look at "split", I can examine 5 dictionaries:

```
split['train']['edge'] 
split['valid']['edge'] 
split['valid']['edge_neg'] 
split['test']['edge'] 
split['test']['edge_neg']
```

I do understand that we need negative edge samples to test the model prediction on negative/non-existing edges as well, but why don't you include negative train edges, i.e. split['train']['edge_neg']? Does this mean that we are not training on negative edges, but only measure the accuracy on negative test and validation links?



Generating negative training edges is expected to be performed inside the model via negative sampling. This models the real case scenario in which we are only given positive relations, and want to infer missing ones.

Thank you very much for the explanation!
Repository: snap-stanford/ogb
Issue: Binary attribute for ogbg-molhiv returns false

The below snippet gives `False`, but `ogbg-molhiv` labels are binary. Unless I'm misunderstanding what the binary attribute represents (I can't find it in the docs) this should evaluate to `True`? I am using ogb version `1.3.1`

```
from ogb.graphproppred import PygGraphPropPredDataset

dataset = PygGraphPropPredDataset(name='ogbg-molhiv', root='dataset')
print(dataset.binary) 
```


Binary attribute just means whether the data file is provided as csv file or numpy’s binary file. For molecule datasets, we provide them as csv file; hence, dataset.binary is False

I see 😳. Is there a page that documents these kinds of things (classes and their attributes etc.)? Is this something that the OGB team is working on?

The best documentation is probably our example code. Those details (e.g., file saving format) are unimportant to the users; hence, we do not provide such documentations.
Repository: snap-stanford/ogb
Issue: Ogbg_molpcba Scaffold Index

Hi, i am working on ogbg_molpcba dataset, and noticed that scaffold index can be downloaded from https://snap.stanford.edu/ogb/data/misc/ogbg_molpcba/. I was wondering how you obtain these scaffold index ? From the source dataset or generate by some software ?  If the later case, can i know how to generate it ? Thanks!

Hi! The code is [here](https://groups.google.com/g/open-graph-benchmark/c/0KML08gNVcM/m/99SeJ0zpAAAJ).

Thanks for your reply. Is the smiles_string stored in mol.csv.gz under ./mapping folder ?  If i want to generate scaffold index from smile_string, is there anything else i need to do ? What i have tried is to open mol.csv.gz file and call scaffold_split function, but it seems the format is not right. i just add these code based on your code send me before:

if __name__ == '__main__':
    with gzip.open("mol.csv.gz", "rb") as f:
        data = f.read()
        train_idx, valid_index, test_idx = scaffold_split(list(data))

The error message:
 mol = Chem.MolFromSmiles(smiles)
TypeError: No registered converter was able to produce a C++ rvalue of type std::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> > from this Python object of type int

Thanks! 





Yes, that's correct. You can read `mol.csv.gz` by 
```python
import pandas as pd
df = pd.read_csv('mol.csv.gz')
smiles_list = df['smiles'].tolist()
```

More details can be found in `mapping/README.md`. Hope this helps!

Thanks! For ogbg_ppa dataset, i was wondering if there is species index, just like ogbg_proteins and ogbl_ppa, species index is included in the dataset ? 

Yes, species index for ogbg-ppa should be in the corresponding `mapping/` directory. See `mapping/README.md` for details.

`mapping/` will be most likely located in `dataset/ogbg_ppa/mapping` 

Hi, OGB Team, for ogbg_proteins and ogbl_ppa, i was wondering how you encode the species index into node features? Just append the species index into each node feature? For each graph, since it only belongs to one species domain, do you encode one species index into all node features within one graph? And for ogbg-ppa, ogbg-molhiv, ogbg-molpcba, do you encode the species index into node or edge feature? Thanks !
Repository: snap-stanford/ogb
Issue: checksum for MAG240M-LSC, WikiKG90M-LSC dataset
Root cause: Unimplemented Feature Gap

Often when we download large dataset, it is best to check the md5sum to validate if the data downloaded is correct. Is it possible to provide checksums of this two datasets and also enhance the download script to incorporate checksum? 

Here is it: https://github.com/snap-stanford/ogb/issues/127

Thanks! closing.
Repository: snap-stanford/ogb
Issue: Submitting new models or datasets to OGB leaderboard
Root cause: Documentation Deficiency

Hi Team,

I'm wondering what is the procedure of submitting new models or datasets to the OGB leaderboard.  Do I simply make pull requests?  How would the leaderboard page be updated?

EDIT: I'm referring to #1 and #9 since I think both issues have the same concern about contributing new datasets to OGB.

Thanks.

Hi, great questions!

For new datasets contribution, please feel free to send us email (ogb@cs.stanford.edu) about your proposal, providing details and why the dataset fits in OGB. We will examine it internally to decide if we will include it. 

For leaderboard submissions, please see [our webpage](https://ogb.stanford.edu/docs/leader_overview). Basically, you submit your result via Google form, and we will update the webpage accordingly.
Repository: snap-stanford/ogb
Issue: Code2 example broken
Root cause: External Dependency Breakage

If you try running the code2 example, there is an out of bounds error. It runs fine on my laptop, which has an old version of the code2 dataset downloaded several months ago. A fresh download does not seem to work.

Here's the error:

```
$ python main_pyg.py
Namespace(batch_size=128, dataset='ogbg-code2', device=0, drop_ratio=0, emb_dim=300, epochs=25, filename='', gnn='gcn-virtual', max_seq_len=5, num_layer=5, num_vocab=5000, num_workers=0, random_split=False)
Target seqence less or equal to 5 is 0.0%.
Traceback (most recent call last):
  File "main_pyg.py", line 270, in <module>
    main()
  File "main_pyg.py", line 176, in main
    vocab2idx, idx2vocab = get_vocab_mapping([dataset.data.y[i] for i in split_idx['train']], args.num_vocab)
  File "main_pyg.py", line 176, in <listcomp>
    vocab2idx, idx2vocab = get_vocab_mapping([dataset.data.y[i] for i in split_idx['train']], args.num_vocab)
IndexError: list index out of range
```

dataset.data.y has length 1, hence why the error is thrown.

Did something change in the version of the dataset uploaded?

Hi! The underlying dataset file did not change, and I obtained the same error as you. This seems to be caused by the change in the collate policy of pytorch geometric (I am using torch_geometric 2.0.1).

```python
data0.y = ['a', 'b']
data1.y = ['c', 'd', 'e']
data_list = [data0, data1]

# after collating and using the pyg dataset object, I got
dataset.data.y = [['a', 'c']]
dataset.data.slices['y'] = [tensor([0,1])]
dataset[0].y = ['a']
dataset[1].y = ['c']

# expected behavior should be
dataset.data.y = [['a', 'b'], ['c', 'd', 'e']]
dataset[0].y = ['a', 'b']
dataset[1].y = ['c', 'd', 'e']
```

@rusty1s Could you please help check this?




Good catch. PyG 2.0 also tries to collate elements of lists (similar to how the standard PyTorch `DataLoader` handles lists), which leads to this change in outcome. I restored the original behavior in PyG for lists which hold elements such as integers and strings. As a result, you can fix this issue by installing PyG from master for now.

Awesome, thanks both!

I might have spoken too soon 🤔

I tried installing PyG from master (i.e. `pip install git+...`) but I still get the same error on the code2 example. The change in behaviour mentioned by @weihua916 is now fixed when I checked in the interpreter when I installed from master, but I think something else is perhaps the source of the issue? The length of `dataset.data.y` is still 1 on PyG master :-(

Ok, let me check that tomorrow:)

I checked once again and the above error is gone for me when using PyG master. Keep in mind that you need to re-process the dataset first. You can simply enforce this by removing the `processed` directory in the dataset folder.

That fixed it! Thanks 🙂
Repository: snap-stanford/ogb
Issue: About node and label information of ogbn-proteins

Hi authors,

Is there a way to know the specifc protein name of each node and the specific protein function name of each label?

Thanks!

Yes, you can!

Go to `${root}/ogbn_proteins/mapping`, and read `README.md` there.

> Yes, you can!
> 
> Go to `${root}/ogbn_proteins/mapping`, and read `README.md` there.

Found it. Thanks!

I see in mapping/README.md, it says the edge feature dimension is 8. However, this is a node classification task, and when I load the dataset, data.x is in shape [N, 8]. Is there a mistake? It should be node feature rather than edge feature.
<img width="545" alt="image" src="https://user-images.githubusercontent.com/56153611/187482185-dc70df20-62da-47f0-925d-ea86059042f1.png">



It is correct. See below.
```
Python 3.8.8 (default, Apr 13 2021, 19:58:26)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from ogb.nodeproppred import PygNodePropPredDataset
>>> dataset = PygNodePropPredDataset(name = 'ogbn-proteins')
>>> dataset[0]
Data(edge_index=[2, 79122504], edge_attr=[79122504, 8], node_species=[132534, 1], y=[132534, 112])
```

My guess is that you apply [this line](https://github.com/snap-stanford/ogb/blob/master/examples/nodeproppred/proteins/gnn.py#L127) to get `data.x`.

Thanks a lot! I have another question about the Protein dataset. I had seen another protein dataset called PPI, with 50-dimension node features processed from C1, C3, and C7 from GSEA database. I am confused with the data pre-processing and tried to contact the contributor of GraphSAGE, which first used this PPI dataset, but do not receive a reply. Do you know any information about this?
Thanks in advance.