zhiminy/EvalEng huggingface dataset:
Repository: snap-stanford/ogb
Issue: [Discussion] Raw text data should be made available for WikiKG90Mv2
Root cause: Unimplemented Feature Gap

I had wanted to start a public discussion but my post got deleted ( #304 ) . So I am reposting as a new issue.

Our method KGT5 (https://github.com/apoorvumang/transformer-kgc and soon to be presented in ACL 2022 main conference) was not included in the leaderboard since it violates the ['no raw data' rule](https://ogb.stanford.edu/docs/lsc/rules/) of OGB-LSC leaderboard submissions. This is because we used the titles + descriptions in text form rather than the MPNet embeddings provided by the ogb library. We used the exact same data as used by the best performing leaderboard method, nothing 'extra'.

Although I understand the restriction on raw data to prevent 'cheating' - which is also the reasoning given in the [discussion here](https://github.com/snap-stanford/ogb/discussions/136#discussioncomment-1769725) - I believe this vastly restricts the space of models that are allowed to feature on the leaderboard. Our model is just as 'fair' as the ones on the leaderboard (such as TransE-concat), and I feel the dataset authors should reconsider the rules for WikiKG90Mv2 since using text embeddings is a big bottleneck and is not ideal for all scenarios.

This is the author's response over email (@weihua916)  when I raised these issues, and asked for test set numbers:

> Great points. Unfortunately, it would be hard for us to make any exceptions. Of course, better textual representation could improve performance, but in OGB-LSC we want to focus on graph methods, and we think the improvement of textual methods and graph methods are somewhat complementary.
> 
> We cannot provide the test set performance. I'd suggest you simply use the validation performance in your paper.

I don't understand the meaning of 'we think the improvement of textual methods and graph methods are somewhat complementary'. In our report, we show our method works on other datasets as well. Also, TransE-concat uses text, so why is that on the leaderboard?

Although I continue to disagree with the leaderboard policies, I want to emphasize that WikiKG90Mv2 is a great resource for training/testing KG Embeddings on a large scale and one of the largest benchmark KGs available with a well thought out test/validation split. I really appreciate the effort put into making the dataset + updating its evaluation scheme in v2. I would request the authors to keep the textual information available - even if not leaderboard worthy - in case someone wants to use this resource in some other form than link prediction.

I hope the authors and those who have contributed to the leaderboard can comment on this. @weihua916 @hyren 


Great suggestion. We understand it and we will definitely think about this option in the future. However, we will not do this at least during the NeurIPS competition. Thanks for understanding.
Repository: snap-stanford/ogb
Issue: ogbn_papers100M load too much data into Ram such that zsh kills the process
Root cause: Resource Mishandling

Hi , I trying to load the ogbn_papers100M dataset in my macbook pro with 32G ram.
After downloading the data, and trying to convert the raw data into ogb data format, Zsh kill the process with the following error message:
'miniconda3/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown'

Is there any  fix or workaround?

<img width="1792" alt="Screen Shot 2022-02-28 at 2 35 40 PM" src="https://user-images.githubusercontent.com/7563707/156047343-084aab5c-7dc7-41e9-96b1-5a746f31c69d.png">



Hi! You will need to use a larger RAM machine to handle such a large graph.
Repository: snap-stanford/ogb
Issue: what does JK stands for in conv.py

https://github.com/snap-stanford/ogb/blob/master/examples/graphproppred/mol/conv.py
```
class GNN_node(torch.nn.Module):
	def __init__(self, num_layer, emb_dim, drop_ratio = 0.5, JK = "last", residual = False, gnn_type = 'gin'):
``` 
I'm sorry to ask this stupid question. but i wonder the full name of JK.
I understood how it works. lt's related to how to return the node_representation.
Thanks in advance

It refers to the usage of **Jumping Knowledge**, see [here](https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html?highlight=jumpingknowledge#torch_geometric.nn.models.JumpingKnowledge).

aha! thanks for the fast answer
Repository: snap-stanford/ogb
Issue: Pretrained models on wikikg90m-v2

Hi, thanks for the large scale dataset wikikg90m-v2.
I would like to explore KGE models trained on large graphs. 
Do you by any chance still have the trained base models TransE and ComplEx? 
Could you share the resulting embeddings?


It would be hard to share the pre-trained embeddings because the size is extremely large (as you could imagine). I suggest you run our script yourself.
Repository: snap-stanford/ogb
Issue: Mismatching between xyz in order and edge index
Root cause: Validation Gap

Hi. Weihua. We found this issue:
for example in an xyz file, the data might be in this format:
C 0 0 0 
C 0 1 1
C 0 2 2
N 1 0 1
Si 1 0 2
with this index order: 0, 1, 2, 3, 4 
It is just an assumption. And we just plotted them in a 3d space with each index. But we found that it might not be matching with the order that we have printed for example C, N, C, C, Si but not C, C, C, N, Si which is corresponding to the edge index. 

We also figure out that the order of C is confused, which means that in the edge index matrix, the index of the first C is 0 but is actually the index 1 in the original xyz file.

We don't know how to deal with it.


Hi! I suggest you use the newly-released SDF format: instructions [here](https://ogb.stanford.edu/docs/lsc/pcqm4mv2/).
Repository: snap-stanford/ogb
Issue: Evaluator for ogbl-collab
Root cause: Validation Gap

Hi, it seems that the example code initializes the evaluator for `ogbl-collab` with `evaluator = Evaluator(name='ogbl-ppa')` [here](https://github.com/snap-stanford/ogb/blob/master/examples/linkproppred/collab/gnn.py#L286).

Thanks! This is fixed now. It's just a typo though since both datasets use the same evaluator metric.
Repository: snap-stanford/ogb
Issue: What is the Experimental setting?

Dear authors, I'd like to know the experimental setting of your experiments in node classification tasks. 
Includes: The number of machines, the Number of GPUs of each machine, and the Memory size of each GPU. (For products, proteins, arxiv, papers100M, and mag datasets respectively)

Hi! We have all the GPU information available in the leaderboard (e.g., [here](https://ogb.stanford.edu/docs/leader_nodeprop/)). 

> Hi! We have all the GPU information available in the leaderboard (e.g., [here](https://ogb.stanford.edu/docs/leader_nodeprop/)).

Thanks for your reply! Were most models (like the model trained by your team) were trained with one machine & one GPU?

Yes. All of our baseline models use a single GPU and one machine.

> Yes. All of our baseline models use a single GPU and one machine.

Thanks a lot~
Repository: snap-stanford/ogb
Issue: ogbn-mag dataset in RDF format

Hi authors, is there a way to download ogbn-mag dataset in any RDF format i.e NT3,TTL, or others.
thanks

We only provide the dataset in the current format. We are not planning to provide it in any other format. Could you please let me know in what case you need the RDF format?

I want to apply RGCN on my custom RDF graph, is there a way to covert my custom graph dataset to the same format like ogbn-MAG.
thanks

I do not think we will have any immediate plan to support RDF graph in OGB. As such, I will close this issue for now. Thanks for the suggestion.
Repository: snap-stanford/ogb
Issue: Training GNN on entire adj instead of training_set_adj
Root cause: Documentation Deficiency

I'm looking at this line of code:
https://github.com/snap-stanford/ogb/blob/c8f0d2aca80a4f885bfd6ad5258ecf1c2d0ac2d9/examples/linkproppred/collab/gnn.py#L107

My question is, why are we training the GNN model on the entire adjacency matrix instead of just the training set's adjacency matrix? If the GNN can see the val and test edges, that information will be incorporated into the GNN's hidden states. Therefore when we send those hidden states to the LinkPredictor to predict presence/absence of edges, wouldn't that be considered data leakage?

Great point. `data.adj_t` only contains training edge information.

@weihua916 Can you please explain how that is so? I am seeing in Lines 221--225 that the edges are not being split when creating the variable `data`:
```    
    dataset = PygLinkPropPredDataset(name='ogbl-collab')
    data = dataset[0]
    edge_index = data.edge_index
    data.edge_weight = data.edge_weight.view(-1).to(torch.float)
    data = T.ToSparseTensor()(data)
```

`data` by default only contain training edges. The val/test edges can be accessed as
```python
split_edge = dataset.get_edge_split()
```

Got it. Thanks!
Repository: snap-stanford/ogb
Issue: One-hot encoding for GIN

GIN assumes that features are one-hot encoded, since summation is injective:
"In the first iteration, we do not need MLPs before summation if input features are one-hot encodings as their summation alone is injective".

Features in graph classification example with GIN are not one-hot encoded, however, while at least some of them easily could be (e.g. atom type). Is this on purpose, and GIN does not show any degraded performance? Also for this reason, shouldn't `AtomEncoder` actually be harmful for GIN performance due to loss of injectivity?

It is one-hot encoded. Atom-encoder uses `torch.nn.Embedding` module, which assumes input to be one-hot encoded (e.g., atom type)

How can they be one-hot encoded, when we have 9 features? "Input node features are 9-dimensional, containing atomic number and chirality, as well as other additional atom features such as formal charge and whether the atom is in the ring or not". I see that first feature is atom number, which is an integer, not a one-hot encoded value. So the model cannot use the raw matrix `x`, since it's not one-hot encoded, and `AtomEncoder` outputs dense vectors with given dimensionality. While using TUDatasets, on the other hand, the `x` matrix would indeed be a huge sparse matrix with one-hot encoded atom types. For OGB I would have to do such encoding myself, using the first column from the 9 features?

Each of 9-dimensional features is an index. Each of the element is embedded independently: https://github.com/snap-stanford/ogb/blob/master/ogb/graphproppred/mol_encoder.py#L22

Why use all embeddings for each feature of the atom/bond? Especially for the cases with true false embeddings where you train an emb_num params for only two classes. Which advantage has to use this than a one-hot encoding of everything + MLP to match the emb_num?
Repository: snap-stanford/ogb
Issue: How can I get SMILES from PCQM4Mdataset?
Root cause: Documentation Deficiency

By inheriting the PCQM4Mdataset, downloading it directly and processing it, I get a pyg Data object as input. Currently, My task needs SMILES to extract other features except the feature defined by smiles2graph in ogb.util. How can I get the original SMILES?

Hi! You can do the following. See more description [here](https://ogb.stanford.edu/docs/lsc/pcqm4mv2/).

```python
from ogb.lsc import PCQM4Mv2Dataset
dataset = PCQM4Mv2Dataset(root = ROOT, only_smiles = True)

# get i-th molecule and its target value (nan for test data)
i = 1234
print(dataset[i]) # ('CC(NCC[C@H]([C@@H]1CCC(=CC1)C)C)C', 6.811009678015001)
```
Repository: snap-stanford/ogb
Issue: RuntimeError while run full_batch.py
Root cause: External Dependency Breakage

Hi,
I run `python full_batch.py` on ogbn_arxiv dataset, while a runtime error occurred. I have tried searching for a solution, but I still don't know why it happened....
Has anyone run into the same problem can give some help? Thank you in advance.

> RuntimeError:
> 
> __init__(__torch__.torch_sparse.storage.SparseStorage self, Tensor? row, Tensor? rowptr, Tensor? col, Tensor? value, (int, int)? sparse_sizes, Tensor? rowcount, Tensor? colptr, Tensor? colcount, Tensor? csr2csc, Tensor? csc2csr, bool is_sorted) -> (None):
> Expected a value of type 'Optional[Tensor]' for argument 'row' but instead found type 'int'.
> :
>   File "/home/Evan/PyEnv_3.7/lib/python3.7/site-packages/torch_sparse/storage.py", line 283
>         col = idx % num_cols
> 
>         return SparseStorage(row=row, rowptr=None, col=col, value=self._value,
>                ~~~~~~~~~~~~~ <--- HERE
>                              sparse_sizes=(num_rows, num_cols), rowcount=None,
>                              colptr=None, colcount=None, csr2csc=None,
> 

Related to https://github.com/rusty1s/pytorch_geometric/issues/1125#issuecomment-634436135 caused by the division op changes introduced in PyTorch 1.5. Please modify `storage.py` accordingly or update to PyTorch 1.5. Sorry for the inconveniences!

I modify `storage.py` and it works now!!!
Thank you for your help :)
Repository: snap-stanford/ogb
Issue: RuntimeError: PytorchStreamReader failed locating file data.pkl: file not found
Root cause: Environment Incompatibility

I am getting a runtime error when I try to download the dataset.
```
import torch
import numpy as np
import torch_geometric.datasets
from ogb.lsc import PCQM4Mv2Dataset
from ogb.utils import smiles2graph
import pyximport

dataset = PCQM4Mv2Dataset(root = 'ogblsc/', smiles2graph = smiles2graph)
```
And the error is:

```
RuntimeError                              Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_3776\3893196584.py in <module>
----> 1 dataset = PCQM4Mv2Dataset(root = 'ogblsc/', smiles2graph = smiles2graph)

C:\ProgramData\Anaconda3\envs\graphormer\lib\site-packages\ogb\lsc\pcqm4mv2.py in __init__(self, root, smiles2graph, only_smiles)
     50             self.prepare_smiles()
     51         else:
---> 52             self.prepare_graph()
     53 
     54     def download(self):

C:\ProgramData\Anaconda3\envs\graphormer\lib\site-packages\ogb\lsc\pcqm4mv2.py in prepare_graph(self)
     80         if osp.exists(pre_processed_file_path):
     81             # if pre-processed file already exists
---> 82             loaded_dict = torch.load(pre_processed_file_path, 'rb')
     83             self.graphs, self.labels = loaded_dict['graphs'], loaded_dict['labels']
     84 

C:\ProgramData\Anaconda3\envs\graphormer\lib\site-packages\torch\serialization.py in load(f, map_location, pickle_module, **pickle_load_args)
    605                     opened_file.seek(orig_position)
    606                     return torch.jit.load(opened_file)
--> 607                 return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
    608         return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
    609 

C:\ProgramData\Anaconda3\envs\graphormer\lib\site-packages\torch\serialization.py in _load(zip_file, map_location, pickle_module, pickle_file, **pickle_load_args)
    876 
    877     # Load the data (which may in turn use `persistent_load` to load tensors)
--> 878     data_file = io.BytesIO(zip_file.get_record(pickle_file))
    879 
    880     unpickler = UnpicklerWrapper(data_file, **pickle_load_args)

RuntimeError: PytorchStreamReader failed locating file data.pkl: file not found
```

Hi! I suspect your .pt file is corrupted (see [here](https://stackoverflow.com/questions/69979034/runtimeerror-pytorchstreamreader-failed-locating-file-data-pkl-file-not-found)). Can you delete the dataset folder and try everything again?
Repository: snap-stanford/ogb
Issue: Process finished with exit code 134
Root cause: Environment Incompatibility

Hi!
When I import ogb package like `from ogb.nodeproppred import DglNodePropPredDataset`, I get an error:
```
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

Process finished with exit code 134 (interrupted by signal 6: SIGABRT)
```
The same happens when I import other packages of ogb including `PygNodePropPredDataset`, etc. 

packages:
PyTorch, 1.11.0
Python, 3.7.10
OGB, 1.3.3

Hi! I do not think the error is related to the ogb package, because our package is written in pure Python. Can you make sure your other packages (such as `torch_geometric` and `torch`) are installed appropriately? For example, when you do `import torch_geometric`, does it run properly?
Repository: snap-stanford/ogb
Issue: [Suggestion] Including SMILES strings for the datasets
Root cause: Unimplemented Feature Gap

Hi,

I noticed it is not possible to modify the data based on the SMILES in some datasets. It would be great to have the smiles2graph parameter in [ogbg-molhiv](https://ogb.stanford.edu/docs/graphprop/#ogbg-mol) (and 10 smaller datasets)  just like in [PCQM4MDataset](https://github.com/snap-stanford/ogb/blob/38a1b223d6f8019f4d2cab5e9445f78e43a791e2/ogb/lsc/pcqm4m.py#L15). 

Thanks,

We have provided the SMILES strings in `mapping/`.

Thank you for the prompt response. The problem with having them in `mapping/` is that since there is no identifier for each sample there is no way to find the corresponding items in the mapping with [pre_transform](https://github.com/snap-stanford/ogb/blob/13f2fe512f3c8d166c2bd4cf02b65f8f3be9ca55/ogb/graphproppred/dataset_pyg.py#L161). Instead, the whole [process function](https://github.com/snap-stanford/ogb/blob/13f2fe512f3c8d166c2bd4cf02b65f8f3be9ca55/ogb/graphproppred/dataset_pyg.py#L120) needs to be modified so that mapping element can be passed to `pre_transform` together with `data`. If one uses this solution to solve this problem, they will need to update the process function manually for upcoming versions. 

So my suggestion is to make mapping information available to pre_transform natively. 

The reason we did not provide SMILES in our dataset object is two-fold:
(1) standardize the molecular graph representation for the graph learning community (not necessarily chemistry experts)
(2) make our package not dependent on rdkit.

i-th molecule in `data_list` is i-th molecule in the file under `mapping`

For the PCQM4Mv2 dataset, our dataset object does provide the SMILES strings.
```python
from ogb.lsc import PCQM4Mv2Dataset
dataset = PCQM4Mv2Dataset(root = ROOT, only_smiles = True)

# get i-th molecule and its target value (nan for test data)
i = 1234
print(dataset[i]) # ('CC(NCC[C@H]([C@@H]1CCC(=CC1)C)C)C', 6.811009678015001)
```
Repository: snap-stanford/ogb
Issue: The problem of loading OGB dataset
Root cause: Validation Gap

When I using the following code (as shown in [4.5 Loading OGB datasets using ogb package — DGL 0.8.1 documentation](https://docs.dgl.ai/en/0.8.x/guide/data-loadogb.html)) to load the OGB dataset,
```
import dgl
import torch
from ogb.graphproppred import DglGraphPropPredDataset
from dgl.dataloading import GraphDataLoader

def _collate_fn(batch):
    graphs = [e[0] for e in batch]
    g = dgl.batch(graphs)
    labels = [e[1] for e in batch]
    labels = torch.stack(labels, 0)
    return g, labels

# load
dataset = DglGraphPropPredDataset(name='ogbg-molhiv')
split_idx = dataset.get_idx_split()
# dataloader
train_loader = GraphDataLoader(dataset[split_idx["train"]], batch_size=32, shuffle=True, collate_fn=_collate_fn)
valid_loader = GraphDataLoader(dataset[split_idx["valid"]], batch_size=32, shuffle=False, collate_fn=_collate_fn)
test_loader = GraphDataLoader(dataset[split_idx["test"]], batch_size=32, shuffle=False, collate_fn=_collate_fn)
```
I have the following problem：
```
python ogb_test.py 
Traceback (most recent call last):
  File "ogb_test.py", line 16, in <module>
    dataset = DglGraphPropPredDataset(name='ogbg-molhiv')
  File "/home/ssy/anaconda3/envs/ELG/lib/python3.7/site-packages/ogb/graphproppred/dataset_dgl.py", line 68, in __init__
    self.pre_process()
  File "/home/ssy/anaconda3/envs/ELG/lib/python3.7/site-packages/ogb/graphproppred/dataset_dgl.py", line 100, in pre_process
    if decide_download(url):
  File "/home/ssy/anaconda3/envs/ELG/lib/python3.7/site-packages/ogb/utils/url.py", line 13, in decide_download
    size = int(d.info()["Content-Length"])/GBFACTOR
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
```
I tracked the bug and found d.info()["Content-Length"]=None. Why did this happen?

Hi! I ran your exact code with `dgl==0.8.1` without any issue. Can you delete the cached dataset folder (e.g., `rm -rf dataset/ogbg_molhiv/`) and run the script again?
Repository: snap-stanford/ogb
Issue: [Suggestion - PyG] option to load heterogeneous graphs as HeteroData 
Root cause: Unimplemented Feature Gap

Hi OGB-Team, 
right now, heterogenous graphs are loaded as a `Data` objects by filling `x_dict`, `edge_index_dict` etc.. `HeteroData` in PyG on the other hand, stores features and edges by its node-type or edge-type identifier, e.g. node-related buffers like `x` are exposed as `data[NODE_TYPE].buffer` and edge-related buffers like `edge_index` are exposed as `data[EDGE_TYPE].buffer`.Those representations are quite similar besides basically the order of keys accessing the data, e.g. `data.x_dict[NODE_TYPE]` vs `data[NODE_TYPE].x`. 

Is there any particular reason for this? If not, `HeteroData` would seem more flexible to me in the PyG framework, as it would allow for direct use with samplers or for transformations like `to_undirected` or `to_homogeneous`. 

This is a good suggestion! For now, it should be no problem to manually convert your data into a `HeteroData` object but I agree that there exists no reason to not integrate PyG 2.0 features within OGB. I will try to do this.

@rusty1s Do you still plan on working on this? Can I pick up this issue?

Just answered you in private. I would really love to see this issue being finally integrated :)
Repository: snap-stanford/ogb
Issue: The ogbn-arxiv dataset is not available?
Root cause: Documentation Deficiency

Dear OGB team, 

I am trying to process the "ogbn-arxiv" dataset, and 
after creating a dataloader like
`dataset = PygGraphPropPredDataset(name = "ogbn-arxiv")`, only to get the following response:
```Traceback (most recent call last):
  File "test_ogb_data.py", line 4, in <module>
    dataset = PygGraphPropPredDataset(name = "ogbn-arxiv")
  File "/home/zhaohuan/anaconda3/lib/python3.7/site-packages/ogb/graphproppred/dataset_pyg.py", line 25, in __init__
    raise ValueError(error_mssg)
ValueError: Invalid dataset name ogbn-arxiv.
Available datasets are as follows:
ogbg-molbace
ogbg-molbbbp
ogbg-molclintox
ogbg-molmuv
ogbg-molpcba
ogbg-molsider
ogbg-moltox21
ogbg-moltoxcast
ogbg-molhiv
ogbg-molesol
ogbg-molfreesolv
ogbg-mollipo
ogbg-molchembl
ogbg-ppa
```

Can you give any suggestion for this problem? Thanks very much.

Hi, that's because you are using graph property prediction dataset object, while `ogbn-arxiv` is a node classification dataset.

You can get the dataset by 
```python
from ogb.nodeproppred import PygNodePropPredDataset
dataset = PygNodePropPredDataset(name = 'ogbn-arxiv')
```

> Hi, that's because you are using graph property prediction dataset object, while `ogbn-arxiv` is a node classification dataset.
> 
> You can get the dataset by
> 
> ```python
> from ogb.nodeproppred import PygNodePropPredDataset
> dataset = PygNodePropPredDataset(name = 'ogbn-arxiv')
> ```

Oh, I see. Thanks very much.