zhiminy/EvalEng huggingface dataset:
Repository: snap-stanford/ogb
Issue: Can‘t achieve reported SIGN performance MAG240M
Root cause: Documentation Deficiency

I ran part of the code locally, but can't achieve as high score as written in the README.md. My local SIGN get 0.65 valid accuracy and SGC only get 0.62. Run the same code but don’t know why

I'm really sorry. I think the committed hyperparameters are slightly wrong, i.e. try using the model with `--layer=2`.

Edit: the default `--layer=3` is correct.

> I'm really sorry. I think the committed hyperparameters are slightly wrong, i.e. try using the model with `--layer=2`.

Thanks for your reply! How about the hyperparameters about gnn.py and rgnn.py? The default hyperparameters can't reach convergence.

Those hyper-parameters should be correct. Could you paste your tensorboard log + any print output + the command you run? It could be also helpful to give us your package version.

I also re-ran the `sgc.py` script (with `--layer=3`) and obtained the same results as reported.

Same as Matthias: I ran `python preprocess_sgc.py; python sign.py --device=0` and it perfectly reproduces the reported SIGN result.

torch==1.7.0, torch_geometric==1.6.3

For GNN, the validation curves on the tensorboard will show lower results than reported, since the sampled neighboring node sizes are small during training. You should be able to reproduce the reported validation result once you run the inference code: `python gnn.py --device=0 --model=graphsage --evaluate`. It uses much larger neighbor sampling sizes.

Closing the issue for now. Let us know if you still cannot reproduce.
Repository: snap-stanford/ogb
Issue: GCN precompute_norm
Root cause: Documentation Deficiency

In the ogb-examples link prediction datasets, I noticed that GCN layer for ogbl-ppa uses precompute_norm and set the normalized=False, cached=False in GCNConv and the GCN layer for other datasets doesn't use the precompute_norm and set the normalized=True, cached=True in GCNConv. Are there any particular reasons for doing this, rather than setting normalized=True, cached=True in GCNConv layer for all datasets?

The `(normalized=True, cached=True)` option will compute and cache normalized edge weights in every GCN layer, which is convenient but only feasible in case such data can still fit into GPU memory. We pre-compute normalization coefficients and set `(normalized=False, cached=False)` for all datasets/models where GPU memory is a major limitation.

That makes sense! Thanks for the clarification.
Repository: snap-stanford/ogb
Issue: Full batch on OGB Products goes out of memory on CUDA
Root cause: Resource Mishandling

When I try running `full_batch.py` in `nodeproppred/products` it throws a CUDA out of memory error.

My PyTorch Geometric version is 1.4.2 and the GPU is V100.

This is to be expected. We could only train it in a full-batch fashion using a GPU with 48 GB. That’s why we also provide scripts that make use of sampling. You can eventually train it in a full-batch fashion on a V100 by reducing the number of layers or the hidden dimensionality.

Got it.  Sorry that I have one more question.

I tried running `full_batch.py` on CPU and it reports 76.87% test accuracy.  What is the hyperparameters to get 78.03 on leaderboard?

I ran with `python3 full_batch.py --use_sage --epochs 400`.

We only tuned the dropout ratio (0.0 and 0.5).
Repository: snap-stanford/ogb
Issue: Detail about ogbl-citation2 subgraph sampling

What is the proper way to sample a subgraph from a large graph like MAG citation graph ?

There is no clear consensus on that one yet for heterogeneous graphs: What we do in our script is to convert the MAG graph into a homogeneous one with distinct edge types, and apply standard sampling techniques on it. If you want your edge types to be uniformly distributed after sampling, you may want to look into the [Heterogeneous Graph Transformer](https://arxiv.org/pdf/2003.01332.pdf) model.

Did the "standard sampling techniques" means snow ball or BFS sampling ?

It means GraphSAGE/ClusterGCN/GraphSAINT sampling---techniques developed for homogeneous graphs
Repository: snap-stanford/ogb
Issue: Bug in RGNN author features preprocessing
Root cause: Resource Mishandling

In the for loop below print("generating author features...") in rgnn.py, ``del inputs`` before ``del outputs`` will yield a segmentation fault if the in_memory option is not turned on.

Can you give a little bit more information? The pre-processing should be independent on whether one uses `--in-memory` or not. And, as far as I see, `del inputs` does not do any harm since outputs are already computed.

Yes, so I was trying to reproduce the RGNN performance on MAG240M dataset, but in the preprocessing stage, I got segmentation fault on the line ``del inputs`` while generating author features.

Since ``del inputs`` is directly after ``outputs = adj_t.matmul(inputs, reduce='mean').numpy()``, I'm guessing it could be the case that ``outputs`` which is the dot product of ``adj_t`` and ``inputs`` is lazily computed and ``del inputs`` would affect the numbers in ``outputs``. (of course I could be totally wrong).

I also tried to backtrace the code with gdb, and here's the backtrace log:

https://pastebin.com/yRrMa4Fx

FYI, my system config is as follows:

python 3.8.8
torch 1.8.1
torch-geometric main branch
numpy 1.19.2

`matmul` is performed directly, it isn't lazy op. I think it is save to delete the inputs afterwards. The segmentation fault might have occurred to a different reason.

Any idea what might be the cause of it? Or would you mind share your exact env config so that I could try to see whether it's a bug in the newest version of torch or numpy or not.

I'm not sure why it does not work for you TBH. You said that it works for you using the `--in_memory` option, but there is no difference regarding pre-processing in these versions. You can also just use the fully-preprocessed node-feature matrix from [here](https://github.com/dmlc/dgl/tree/master/examples/pytorch/ogb_lsc/MAG240M).

My config is:
```
pytorch-lightning==1.2.0rc1
pytorch==1.7.1
torch-geometric==1.7.0
numpy==1.20.1
```

Piggybacking on this by noting that I have also gotten seg faults/ core dumps but in this case when trying to actually use the `--in-memory` option on the `gnn.py` based models SAGE and GAT i.e. if `--in-memory` then fault. I will follow up with a trace/dump if I can. 

I was wondering whether it might be because of `torch>1.7` like @hengdashi and I both have

I may try downgrading my venv, it's just finnicky to get PyG,PyTorch,OGB, and CUDA to all agree so I stopped at first working which was `torch==1.8+cu111`

env:
```
pip list

torch                        1.8.0+cu111
torch-geometric                    1.7.0
torch-scatter                      2.0.6
torch-sparse                       0.6.9
...
nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Tue_Sep_15_19:10:02_PDT_2020
Cuda compilation tools, release 11.1, V11.1.74
Build cuda_11.1.TC455_06.29069683_0
...
Python 3.6.10 |Anaconda, Inc.| (default, May  8 2020, 02:54:21) 
[GCC 7.3.0] on linux
>>>
```
Repository: snap-stanford/ogb
Issue: Load smaller dataset
Root cause: Unimplemented Feature Gap

Hi,

Is there a way to load a smaller dataset? I'm trying to run a sample test on my machine to ensure my code works and then move it to the server, but my machine only has 16GB of memory which is not enough apparently, and the python script end up being killed by the OS. I tried looking at the docs and the source code, but there doesn't seem to be anything.

```
$ /usr/bin/time -v python test.py
Loading necessary files...
Command terminated by signal 9
	Command being timed: "python test.py"
	User time (seconds): 155.79
	System time (seconds): 23.27
	Percent of CPU this job got: 96%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 3:05.40
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 12009284
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 1085
	Minor (reclaiming a frame) page faults: 2240431
	Voluntary context switches: 3797
	Involuntary context switches: 6358
	Swaps: 0
	File system inputs: 2584032
	File system outputs: 0
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

```

Hi. Although our current datasets are all medium-scale (around 1M nodes), we are currently preparing small-scale OGB datasets (around 100K nodes) that are easier to be dealt with. Stay tuned. 

We have added a small dataset called ogbn-arxiv. We will add more small-scale datasets in the next release. Stay tuned!
Repository: snap-stanford/ogb
Issue: About the implementation of JK in code of ogbg-mol

Hello, thanks for the code.
In the code [https://github.com/snap-stanford/ogb/blob/master/examples/graphproppred/mol/conv.py](url), the implementation of JK is trying to add the first five items in h_list:
```
elif self.JK == "sum":
      node_representation = 0  
            for layer in range(self.num_layer):  
                 node_representation += h_list[layer]
```
It seems that the output of last layer does not be added to the node_representation.
So, should the output of the last layer also be added to node_represetation ? 

Thanks, good point! `self.num_layer` should be `self.num_layer + 1`
I have fixed the example.

Thanks.
Repository: snap-stanford/ogb
Issue: Has anyone reproduced the results of ogb_lsc_MAG240M baseline ?
Root cause: Environment Incompatibility

Has anyone reproduced the results of this experiment?
I use the same dataset and the same code with [this](https://github.com/dmlc/dgl/tree/master/examples/pytorch/ogb_lsc/MAG240M)
It is reported the validate accuracy is 0.701. But I got it that validate accuracy 0.48.
Is there anything I should do with the original code?

`
Initializing data loader...
Loading model...
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 8685/8685 [20:21<00:00,  7.11it/s]
  0%|                                                                                                                 | 0/9177 [00:00<?, ?it/s]Validation accuracy: 0.4814500284276965
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 9177/9177 [18:57<00:00,  8.07it/s]
`


I guess that your full feature matrix is corrupted to some extend. You can alternatively download it from [here](https://dgl-data.s3-accelerate.amazonaws.com/dataset/OGB-LSC/full_feat.npy) to see if that fixes your issue.

Since you are using the DGL version, it might be a good idea to directly ask for help there - this example is directly maintained by the DGL team.

Hi rusty1s
The md5 of full_feature.npy I used is d98ffac92986de2fdaabc3fe44ced36c.
I download the file from your link, and the same md5.
So the full feature matrix is not corrupted.

Is there any other problem?

Could you please ask this to DGL Team, as they are maintaining the code?

Are you sure the model is done with training? It looks like 0.48 accuracy is obtained after the first epoch.
Repository: snap-stanford/ogb
Issue:  memory error
Root cause: Resource Mishandling

Hello, when I run the file: ogb-master/examples/lsc/mag240m/gnn.py, this error （RuntimeError: [enforce fail at CPUAllocator.cpp:67] . DefaultCPUAllocator: can't allocate memory）appears when converting the adjacency matrix（gnn.py line 61）
I want to ask, is there some way to run this code while reducing memory usage? For example, is it possible to run on 64G memory? 
Thank you very much!

See https://github.com/snap-stanford/ogb/discussions/121
Repository: snap-stanford/ogb
Issue: More protein information for ogbl-ppa

Hi OGB team,

Thank you for the great work. I'm working on ogbl-ppa and I'm wondering if you can share informations (such as protein display name and sequences, etc.) of proteins used in constructing ogbl-ppa? In this way, the dataset will be interesting to a broader community like people in bioinformatics.

Hi, thanks for your interest. That is definitely on our todo-list: to provide meta-information on each dataset to connect to broader context. For now, we are focusing on including more datasets and benchmarking models. Please allow us some time to reach to your request... Thanks for understanding!

This is now addressed. Each dataset has `mapping/` directory that contains information to map nodes/edges/graphs into real-world entities. In `ogbl-ppa`, for example, we map each node into a unique identifier of a protein that is commonly used in biology.

To obtain the most recent dataset, please first delete the existing folder, and rerun the script to re-download the dataset.

With ogb version 1.1.1, you no longer need to manually delete the dataset folder :) The package will handle dataset updates automatically.

Hope we have addressed your comment.
Repository: snap-stanford/ogb
Issue: About the GINE and DeeperGCN's performance on PCQM4M-LSC dataset.

Hi, Weihua and anyone who is interested in this track!

Can anyone share the performance of GINE and DeeperGCN on PCQM4M-LSC dataset.
I believe these two models are good baselines for this track in addition to GIN and GCN models.

Thanks in advance for anyone' share.
Best,
Chengxuan

Hi! This is not really an issue of the code. I do not have a number for DeeperGCN. GINE is essentially GIN (with edge feature).

If you want to solicit information from the community, the discussion forum would be the best place: https://github.com/snap-stanford/ogb/discussions/categories/pcqm4m-lsc
Repository: snap-stanford/ogb
Issue: Platform-agnostic data loaders?
Root cause: Unimplemented Feature Gap

Are there plans for providing a vanilla numpy array loading mechanism for these datasets? This would open up usage of OGB for platforms like TensorFlow or JAX.

Thank you for your interest. This is a good point. We are preparing unified csv format to store graph data that does not depend on specific deep learning libraries.
It will look something like this: https://github.com/snap-stanford/ogb/tree/master/ogb/io
We are working on adapting this format to graph property prediction datasets, where we have many isolated graphs.

Glad to hear that! I'm somewhat confused by the issue closing though - to me that means the issue is either fixed or that there's no intention of fixing. If it's not either of those, could you reopen the issue?

I am excited to let you know that our new release of OGB package will have the platform-agnostic data loaders. Please stay tuned. Thanks!
Repository: snap-stanford/ogb
Issue: Edge attributes of ogbg-ppa
Root cause: Documentation Deficiency

Hi OGB team,

May I ask if the edge attributes of ogbg-ppa dataset are exactly the same as the protein dataset used in [STRATEGIES FOR PRE-TRAINING GRAPH NEURAL
NETWORKS](https://arxiv.org/pdf/1905.12265.pdf#page=15&zoom=100,110,256)? 
![image](https://user-images.githubusercontent.com/76140761/117667759-d126d480-b1d7-11eb-83d8-e97eaffd650a.png)

Thank you!

Hi! No, they are not exactly the same. I believe the dataset source is the same but ogbg-ppa has float features representing the confidence scores of each protein-protein association.
Repository: snap-stanford/ogb
Issue: MAG240M: The difference in accuracy of rgnn.py depending on the batch size when evaluating
Root cause: External Dependency Breakage

In rgnn.py, I reproduced the accuracy described in README.md (70.48%) for the validation data.
```
Testing: 100%|███████████████████████████████████████████████████████████████████| 8685/8685 [10:28<00:00, 13.82it/s]
--------------------------------------------------------------------------------
DATALOADER:0 TEST RESULTS
{'test_acc': 0.7048651576042175}`
```

On the other hand, I noticed that if I change the hard-coded batch_size in line 433 to speed up the operation, the accuracy changes.
For example, when I set batch_size=1024, the accuracy decreased.
```
DATALOADER:0 TEST RESULTS
{'test_acc': 0.6956417560577393}
```

On the other hand, if I set batch_size=1, the accuracy increased.
```
DATALOADER:0 TEST RESULTS
{'test_acc': 0.7176515460014343}
```

I wondered if the remainder of the data divided into batches was being truncated, but even if 1,000 pieces of data were being truncated, I wondered why the accuracy would change by 1% for over 130,000 validation data.

In addition, I tried to output y_pred using `save_test_submission()` in mag240m.py, but it was almost the same regardless of batchsize.

Why does changing the `datamodule.batch_size` affect the accuracy? Is the accuracy calculated by sampling only a portion of the validation data, and does it vary with batchsize?


Hi! It is because of the bug in pytorch-lightning in computing accuracy: see https://github.com/snap-stanford/ogb/discussions/141#discussioncomment-584011 and https://github.com/PyTorchLightning/pytorch-lightning/issues/6889
I believe batch_size=1 gives you the most accurate result, but you may need to re-implement the evaluation code until the bug in pytorch-lightning is fixed.



Thanks for the answer, and sorry it was already mentioned in the discussion.
I was also trying to turn `datamodule.sizes` to a ridiculously large value like him to eliminate uncertainty. However, I was having trouble reproducing the high accuracy of 71.7% that `trainer.test` outputs when `batchsize=1`, when I recalculated using the output file.
As it is a bug in pytorch-lightning, I'm going to trust the accuracy of my own calculations.

Yes, I believe the correct code should give you 71.7%. Let us know if this is the case.

Actually, there seems to be a easy workaround: https://github.com/PyTorchLightning/pytorch-lightning/issues/6889#issuecomment-830234986

I will take care of adding the workaround to our examples.

This is now fixed in the example scripts. We will update the validation accuracy soon.

The validation accuracy and test accuracy have been updated with the new code: https://github.com/snap-stanford/ogb/blob/master/examples/lsc/mag240m/README.md
Repository: snap-stanford/ogb
Issue: About Leaderboard Submission
Root cause: Documentation Deficiency

Hi OGB team,

For leaderboard submission, will you also consider submissions accompanied with a short report rather than a paper? There can be cases where people develop models based on existing approaches with some slight modification for the particular dataset. In such cases, there might not be enough novelty for a full paper.

Hi, that's a great question. Yes, for now, short report is also fine, but we emphasize that we welcome research-style submission rather than Kaggle-style submission.

To follow-up, if the submission is mostly based on an existing model with minor modification to adapt the model to a particular dataset, one can simply use the original paper as a reference and make an "unofficial" leaderboard submission.

Please let us know if you have any feedback or suggestions!

Got it, thanks.
Repository: snap-stanford/ogb
Issue: BIO KG Dataset has mislabels?
Root cause: Documentation Deficiency

It appears that the same index values have multiple different types.

For example, if we take head index value 0, then search for all the instances in the training set. 

The head types have the following distribution:
function    23
protein      2
disease      1
drug         1

Could you confirm that this is indeed an error in the dataset?

See Jupyter Notebook Here:
https://colab.research.google.com/drive/1pUWrZVLve4Ohc3w3ZmsPYAIy55_T4NIC?usp=sharing

![image](https://user-images.githubusercontent.com/40600201/118318761-1dd11f00-b4fa-11eb-877e-31eea44e206e.png)


Hi! This is not an error.
We index different node types differently. In other words, 0-th drug is different from 0-th function.

You can think of `(node_type, idx)` to specify a distinct entity in the biomedical KG.

Ok, thanks now it's much clearer
Repository: snap-stanford/ogb
Issue: Negative samples don't have a type
Root cause: Documentation Deficiency

As we discussed, the entities are differentiated by both (type, index) and not just (index) in #210

However, in the negative samples of the **valid set**, there is no specification on what type it is.

If we assume, it is the same as the positive sample,  it creates more than the entities specified (9377).

![image](https://user-images.githubusercontent.com/40600201/118352409-97e8be80-b561-11eb-9cb1-b471dc59a60e.png)

Notebook:
https://colab.research.google.com/drive/1pUWrZVLve4Ohc3w3ZmsPYAIy55_T4NIC?usp=sharing



Kindly let us know if you want us to edit the valid and train dataset to be coherent with the (index, type) data format. 

(Earlier the better)

Here is the answer: https://ogb.stanford.edu/docs/linkprop/#ogbl-biokg
"we only consider ranking against entities of the same type. For instance, when corrupting head entities of the protein type, we only consider negative protein entities."

That makes sense, however it felt misleading. Maybe we can push a version that is changes the indices as implemented in this notebook:
https://github.com/BrianPulfer/GDL-FinalProject/blob/main/notebooks/ComplEX_best_performances.ipynb

I think the current format is the best way to represent the heterogeneous KG, and users should feel free to convert it to a homogeneous KG if they want (which would inevitably drop node type information). Therefore, we will keep the format as it is.

Also, in terms of a method, it does not make much sense to perform negative sampling from invalid node types, e.g., training against protein tail nodes even if we know the tail type should be disease node.
Repository: snap-stanford/ogb
Issue: Question regarding the link prediction datasets

Hi, I'm a bit confused about the relationship between the training edge set and the graph adjacency we load for link tasks.

My understanding is that, the two are disjoint sets. The adjacency is for you to do message passing, while the training edges is for you to make predictions and compute loss. I have no trouble with all of the full-batch examples, but when I look at the code below:
https://github.com/snap-stanford/ogb/blob/048ef636fda3d0c4a25b108651bf1d43050fa7ae/examples/linkproppred/citation2/cluster_gcn.py#L93

It seems that you directly compute losses on the edges from the adjacency you have done message passing on (the train function does not use the split_edge['train'] at all). Is this an analogy of the training edges, or the adjacency and the training edges are identical (I checked the shapes of the two it seemed that they are not identical)? 

Looking forward to your reply!

Hi! Great point. `data.edge_index` is the same as `split_edge['train']` and *not* disjoint (`data.edge_index` is made undirected to allow bi-directional message passing for undirected graph datasets). In our example code, we indeed perform message passing over `data.edge_index` and try to predict `split_edge['train']`. 

If you want to perform message passing over a subset of training edges, you need to explicitly write that code.

Thanks for the prompt reply! That's crystal clear!
Repository: snap-stanford/ogb
Issue: Regarding the train/validation splits
Root cause: Documentation Deficiency

Hi,

I was wondering if you have trained the networks over a fixed train and validation sets for 10 times, and not 10 different splits.

Looking forward to your reply, and thanks.



That's what we did. We have a single official split, over which we trained 10 models with 10 different random seeds.

thank you for the fast reply. :)