zhiminy/EvalEng huggingface dataset:
Repository: snap-stanford/ogb
Issue: problem about graph feature

    I want to konw what  "possible_degree_list"  mean, I guess it means the node degree in the molecular
graph, but after I check that, not at all. could please help me ?
<img width="553" alt="微信图片_20201012092754" src="https://user-images.githubusercontent.com/32509736/95696335-3da31a80-0c6d-11eb-960a-b875169fc24a.png">
 

According to the [rdkit document](https://www.rdkit.org/docs/source/rdkit.Chem.rdchem.html), `GetTotalDegree()` returns the degree of the atom in the molecule including Hs. The degree of an atom is defined to be its number of directly-bonded neighbors. The degree is independent of bond orders.

Closing the issue for now. Feel free to re-open if you have further questions.
Repository: snap-stanford/ogb
Issue: Updating paper100M dataset seems very time consuming.
Root cause: Environment Incompatibility

Hi OBG Team, hope you are doing great! I'm currently working on the `paper100M dataset` while it seems to spend so much time to update. It has only finished 8% for taking 15 hours. Any idea about this?
![image](https://user-images.githubusercontent.com/29864861/95745618-157fdf80-0c9e-11eb-9492-3ca5481e3712.png)


Hi! It is likely due to your slow internet connectivity. The compressed data size is actually slightly smaller than the previous version. 

Besides, the graph dataset itself has not been changed. This update is to address the issue https://github.com/snap-stanford/ogb/issues/46.

I see! Thank you very much for your reply and clarification.
Repository: snap-stanford/ogb
Issue: papers100M dataset [ zipfile.BadZipFile: File is not a zip file]
Root cause: Environment Incompatibility

Hi OGB Team, hope you are doing great. There is an error during downloading the papers100M dataset (please kindly check out the screenshot below). 
![error](https://user-images.githubusercontent.com/29864861/95851876-8f29d300-0d5b-11eb-81b0-6161bf869911.jpg)

The versions of relevant libraries in the  conda env I'm using are:
Python == 3.7.7
PyTorch == 1.5.0
torch-geometric == 1.6.0
Numpy == 1.18.1
pandas == 1.0.3
urllib3 == 1.25.9
scikit-learn == 0.23.1

which are perfectly satisfying what ogb package requires:
Python>=3.6
PyTorch>=1.2
DGL>=0.5.0 or torch-geometric>=1.6.0
Numpy>=1.16.0
pandas>=0.24.0
urllib3>=1.24.0
scikit-learn>=0.20.0
outdated>=0.2.0

Any ideas about this issue? Thank you for your help in advance!


Hi! Most likely, data files are screwed. Could you show me your downloaded directory? Simplest thing to try out is to clean up your downloaded directory and run your script again (it would re-download the dataset though..)

P.S.: I tried constructing papers100M from scratch on my end, and it worked without the zip issue.
<img width="1407" alt="スクリーンショット 2020-10-13 14 06 22" src="https://user-images.githubusercontent.com/16249234/95916391-67d40580-0d5d-11eb-8618-5b9f3880fec9.png">


Hi, thanks for your reply. The dataset directory contains two folders `ogbn_papers100M` and  `papers100M-bin.zip`. The folder named `raw `in `ogbn_papers100M` is empty. I already emptied the previously downloaded dataset before I ran my script. I will try again as you suggested and figure out what happened. Many Thanks.
![image](https://user-images.githubusercontent.com/29864861/95915178-764b0d00-0daf-11eb-902b-bd47d6b6973a.png)



Hi, I just found out the issue is caused by the limited storage space. Many thanks!

> Hi, I just found out the issue is caused by the limited storage space. Many thanks!

How large the total files are?
Repository: snap-stanford/ogb
Issue: GNNs for heterogeneous link prediction datasets?

Is there a specific reason (or even several?) why you did not consider some GNN (e.g., R-GCN) with ogbl-biokg etc.? 
I am wondering if we have to consider something specific when experimenting with new GNNs.

Hi! Please feel free to try R-GCN. We did not try it because ogbl-biokg does not come with node features.
Repository: snap-stanford/ogb
Issue: Enable access to different splits
Root cause: Unimplemented Feature Gap

Hi,

The dataset code provides a split_type argument in the get_idx_split() method. However, it seems that providing any value to this argument triggers an error because the data found at the provided url only provides the default OGB split index. It would be great if the user were allowed to specify different splits (by adding the different indices to the url). While I understand using a fixed split for benchmarking purposes is necessary, it may be useful for general use of the library and for debugging to experiment with different splits. 

In addition, the scaffold splits used for all MoleculeNet datasets do not match with the splits used by default by MoleculeNet. Thus enabling different splits would be particularly useful to compare with reported scores on these datasets.

Hi! If you want to use the different splits, I'd suggest you generate the indices by yourself, rather than calling `get_idx_split()`.  

> In addition, the scaffold splits used for all MoleculeNet datasets do not match with the splits used by default by 
> MoleculeNet. Thus enabling different splits would be particularly useful to compare with reported scores on these datasets.

That could be the case, although our scaffold implementation followed closely to that of MoleculeNet.  Could you show evidence that ours and MoleculeNet's scaffold splits are different?

For the second point, I only meant that Moleculenet's results are usually not reported on the scaffold split (not that the two scaffold splits necessarily differ), so until OGB becomes a standard, it may be useful to have other splits available for more direct comparison.

I can generate the splits myself, however I am raising the issue because the support for this in the code is already there, and I think it is a relatively useful feature for the package to make this process standardized across splits. 

I see what you mean. Thanks for clarifying. The feature is there in case we want to provide a different official split in the future (e.g., we find a significantly better split, or we decide to evaluate models on multiple splits instead of one). The feature is mainly for our use, and we do not encourage the custom splits to be deeply built into the package.

I see. Thank you for your quick response. I'll close the issue in that case.

I do think it would be worth considering multi-split evaluation in the future and perhaps adding MoleculeNet's splits to the official splits.
Repository: snap-stanford/ogb
Issue: Negative sampling during training for link prediction

Hi OGB team, Quick Q. on negative sampling during training for DDI vs COLLAB: Is there a fundamental difference between the two approaches? Can we use the COLLAB approach to negative sampling in DDI, too?

COLLAB:
https://github.com/snap-stanford/ogb/blob/330a3515f8677464743745d2abe336592140a948/examples/linkproppred/collab/gnn.py#L113

DDI:
https://github.com/snap-stanford/ogb/blob/330a3515f8677464743745d2abe336592140a948/examples/linkproppred/ddi/gnn.py#L118

Hi! Yes you definitely can. We used strict negative sampling for ogbl-ddi because ogbl-ddi is really a dense graph; hence, there will be lots of false positives in your negative samples if you do random negative sampling.

Besides, please be aware of the slight difference of rules for the ogbl-collab: https://ogb.stanford.edu/docs/leader_rules/



Thanks for the prompt response!
Repository: snap-stanford/ogb
Issue: Some questions about ogbn-arxiv dataset

Hello!
I found some papers can cite newer papers on ogbn-arxiv dataset (e.g. a paper published in 2013 cites another paper published in 2014). Why is this happening?
Thanks in advance.


Hi! Good point! This is due to the date gap between pre-print versions and official publications.

I noticed that some are even 20 years apart.
However, in the prediction task, we are more concerned about the year of the pre-print version and the publication year has no practical use.

Interesting. You can look into the actual MAG webpage using our mapping to see what is happening. 

For arxiv papers, the years should be the first time the papers appeared on arxiv; so the task has practical value. MAG keeps updating the citations after those papers are updated on arxiv, resulting in old arxiv papers citing new ones. We have decided to retain those edges rather than removing them because they are pretty rare (around 2%) and that's how real data looks like.
Repository: snap-stanford/ogb
Issue: A question about ogbg-ppa
Root cause: Resource Mishandling

I have 64 GB memory in my Ubuntu server, but I can not process ogbg-ppa data set normally. The memory will overflow when saving the data set into PyG object, and then the process is killed. How much momery do I need in the computer to process this dataset? Do I have other choice to reproduce the experiment on the dataset on 64 GB memory?

Hi! It is a big dataset, and you will need more memory. I have not tested exactly how much memory you need, but I'd assume 100GB should be enough.

Edit: The saved dataset is around 30GB, but extracting compressed files and pre-processing graphs seem to consume more memory than that.
Repository: snap-stanford/ogb
Issue: Persistent issues in the website

I don't know what changed, but in the past couple days I've often been running into a lot of website issues. The page will just load a blank screen.

It comes and goes, but I've ran into it several times (and so have my collaborators)

Perhaps it's because I'm using adblock?

Hi! Yes, I have observed a similar issue recently but have not identified the cause yet. As the issue is sporadic and no fundamental changes have been made on our webpage, it could be related to Github page servers (they are sometimes unstable). 

P.S. Firefox seems to be working fine.

Closing this issue for now. Let me know if the issue still persists annoyingly. Google chrome is working fine recently on my end.

It's definitely still persisting. It's very intermittent - sometimes it'll go away after 5 minutes. For example, right now, node and link leaderboards work, but graph doesn't.

Got it. Does that happen on specific pages (I often see the issue at the dataset overview and team pages)? Also, what if you change the browser?

The pages I check the most are node/link/graph property prediction pages, so those are the ones I notice are most often broken. It doesn't seem to be a browser specific issue - usually when the issue happens I have the same issue on both chrome and firefox.

Got it, thanks for the feedback.
Repository: snap-stanford/ogb
Issue: Clarification about #edges in ogbl-collab

Hi, 

Just looking for some clarification about the link prediction datasets. Per the example given on the website, for ogbl-collab, when I run `dataset[0]` I get 2358104 edges. This is more than what is stated in the description, yet this is stated to be the training edges (which it doesn't match with `train_edge['edge']`. Can you explain this? 


Hi! What is on the website is the number of edges in train+validation+test sets (see the description of Table 3 in [our paper](https://arxiv.org/pdf/2005.00687.pdf)).

I see, but then what is `dataset[0]` supposed to represent? Why are there more edges here (2358104) than in the test + training + validation?

```
>>> train_edge['edge'].size()
torch.Size([1179052, 2])
>>> train_edge['edge'].size()
KeyboardInterrupt
>>> valid_edge['edge'].size()
torch.Size([60084, 2])
>>> test_edge['edge'].size()
torch.Size([46329, 2])
>>> dataset[0].edge_index.size()
torch.Size([2, 2358104])
```

Does `dataset[0]` include each edge twice (with alternating start/end nodes) since it is an undirected graph? 

Yes, that's correct. For all undirected graphs, we represent them as graphs with bidirectional edges. See **Note** [here](https://ogb.stanford.edu/docs/linkprop/).
Repository: snap-stanford/ogb
Issue: edge split for link prediction codes
Root cause: Documentation Deficiency

I have a question regarding edge data split in link prediction codes(https://github.com/snap-stanford/ogb/blob/9cbd598a31cb7934e69827eaccccc69135b8ef69/examples/linkproppred/citation/sampler.py).

To create both PositiveLinkNeighborSampler and NegativeLinkNeighborSampler, "edge_index" from PygLinkPropPredDataset is used, which I assume is the whole edge dataset. Isn't there a need to separate edge pairs for train, validation, and test sets to evaluate based on untrained edge pairs(val/test)? Pardon me if there is any other preprocessed done on the data that I was unaware of.

Hi! `edge_index` of our graph object always only contains training edges.
Repository: snap-stanford/ogb
Issue: cannot import name 'PygLinkPropPredDataset' from 'ogb.linkproppred'
Root cause: Environment Incompatibility

Hi OGB team,

The following import error occurs, when I tried to import PygLinkPropPredDataset.

>>> from ogb.linkproppred import PygLinkPropPredDataset
Using backend: pytorch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'PygLinkPropPredDataset' from 'ogb.linkproppred' (/home/user/miniconda3/envs/DGLpy38/lib/python3.8/site-packages/ogb/linkproppred/__init__.py)

But, it is OK to import Evaluator, e.g.,
>>> from ogb.linkproppred import Evaluator

Hi! Have you installed Pytorch Geometric?

> Hi! Have you installed Pytorch Geometric?

It works. Thank you for your help.
Repository: snap-stanford/ogb
Issue: Proposal for larger data sets
Root cause: Unimplemented Feature Gap

Most of the graph classification datasets seems quite small. Have you looked at

(1) https://arxiv.org/abs/1906.09427
(2) https://arxiv.org/abs/1904.06046 (221 million molecules) ?



Thanks for the reference. We are aware of both of them, and they are definitely interesting. For now, we are focusing on our first release of OGB; after that, we can discuss whether to include them in our OGB datasets. Thanks!

Thanks for the suggestion. The datasets are interesting, but we already have many molecule datasets, and we are not planning to add more of them for now.
Repository: snap-stanford/ogb
Issue: ImportError: cannot import name 'DglNodePropPredDataset' from 'ogb.nodeproppred'
Root cause: Environment Incompatibility

ImportError: cannot import name 'DglNodePropPredDataset' from 'ogb.nodeproppred'

Version: 1.2.3


Sorry, I haven't install dgl before ogb.

can you tell me how to solve the problem?thank you very much!!!!
Repository: snap-stanford/ogb
Issue: role of "root_emb" layer in GCN examples for graph property prediction task

Hi,

I am a bit confused by your implementation of GCN in the examples for the graph property prediction task. 
For example, in `ogb/examples/graphproppred/mol/conv.py`, you wrote GCN message-passing as
```python
self.propagate(edge_index, x=x, edge_attr = edge_embedding, norm=norm) + F.relu(x + self.root_emb.weight) * 1./deg.view(-1,1)
``` 
I don't quite understand the role of `self.root_emb` in this line of code. To me, this layer seems to be of the same function as the bias in the previous linear transform of `x`... Could you please explain in more details why this embedding layer is useful here? Is it a way to improve the performance of GCN?

Thank you!


Hi! You can think of `root_emb` as the self-edge feature (connecting from the node to the same node). We just replaced `edge_attr` with `root_emb` [here](https://github.com/snap-stanford/ogb/blob/master/examples/graphproppred/mol/conv.py#L61).

I have not tested other modeling choices, but I do not think this is so essential to the model performance.

Now I see... Thanks a lot for the explanation!
Repository: snap-stanford/ogb
Issue: Leaderboard for Node Clustering Task?

Hi, Thank you for this awesome work.  I have a question regarding Clustering task. I am working on an Unsupervised(Auto-encoder) based GNN method, Most of the paper in this category are using Cora, Citeseer, Wiki etc. [[1]](https://www.ijcai.org/Proceedings/2018/0362.pdf) [[2]](https://arxiv.org/pdf/1902.08813.pdf) [[3]](https://arxiv.org/pdf/1908.02441.pdf)

If I am using OGB datasets, which datasets I can use for clustering task and is there any leaderboard for that?

Thank you!
Aaditya

Hi! Thanks for your interest. 

Please feel free to use our node classification datasets for evaluating your node clustering algorithms. Regarding leaderboards, we are not planning to provide ones that are specific to node clustering.
Repository: snap-stanford/ogb
Issue: why pass single batches

Hi,
Thank you for the great library.

I have a question concerning the example you provided for predicting the graph property.
In fact, in the train function https://github.com/snap-stanford/ogb/blob/da100e07c5c9e7c3be72ee1e13738f3f7992bb5b/examples/graphproppred/mol/main_pyg.py#L24 you skip the batch if it's a single batch `batch.batch[-1] == 0`. What is the reason for that?

Thank you in advance!

Hi! see this issue: https://github.com/snap-stanford/ogb/issues/67

this answer the question.

Closing the issue!
Repository: snap-stanford/ogb
Issue: Make path to store data configurable
Root cause: Documentation Deficiency

When running OGB to fetch data, it will by default fetch the data and store it to `./dataset`.

Maybe storing the data globally at `~/.ogb`can help save disk space and make it more convenient.
Thanks!

Hi! Thanks for asking this. You can configure the path as follows.
```python
from ogb.nodeproppred import NodePropPredDataset
dataset = NodePropPredDataset(name = 'ogbn-arxiv', root = '~/.ogb')
```

At this point, we may not change the current default path to the global path since existing users may inadvertently re-download the datasets.

I have updated the website to give brief instruction: https://ogb.stanford.edu/docs/home/#dataloader

Great help, thanks!
Repository: snap-stanford/ogb
Issue: minor change to the PCBA benchmark
Root cause: Documentation Deficiency

The method GINE+, reported in the mol-pcba with a score of 0.292, uses virtual node. So I am suggesting to change the name to `GINE+ + virtual node`. The results without virtual node are not reported in the original [paper](https://arxiv.org/pdf/2011.15069.pdf).

Thank you

Sounds good. I have modified it as suggested.
Repository: stanford-crfm/helm
Issue: Display description field in `training_efficiency.json` on website to provide an explanation for various training efficiency metrics
Root cause: Unimplemented Feature Gap


@deepakn94 Could you take this or explain what needs to be done?

I have no idea how any of the frontend code works, so might be faster / easier if someone who has worked with that before does it.

What needs to be done is simple. The "description" field in the training_efficiency.json file (https://github.com/stanford-crfm/helm/blob/main/src/benchmark/efficiency_data/training_efficiency.json#L5) needs to be displayed on the frontend (ideally while mousing over the respective field in a view like this: https://nlp.stanford.edu/helm/v8/?suite=v8&group=synthetic_efficiency). Or maybe we add a "reason" field in the table and add it there.

This isn't done in the frontend, but in `summarize.py` in `create_cell` - a cell has a description field that needs to be set when you mouseover it.

Ok, I can take a stab at it.

Any updates on this?

Will get to it this week.

training_efficiency is deprecated for the Lite leaderboard.
Repository: stanford-crfm/helm
Issue: sqlite3.OperationalError: unable to open database file
Root cause: Environment Incompatibility

I'm seeing this error now, which causes the process to hang:

```
Exception in thread Thread-28865:
Traceback (most recent call last):?it/s]
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/site-packages/sqlitedict.py", line 395, in run
    conn = sqlite3.connect(self.filename, check_same_thread=False)
sqlite3.OperationalError: unable to open database file
```

@yifanmai's work to migrate to Redis should solve this issue.

Related to #944
Repository: stanford-crfm/helm
Issue: Presentation framework v2
Root cause: Unimplemented Feature Gap

Better frontend for https://nlp.stanford.edu/crfm/benchmarking/status.txt which was created for #74  

 If these JSON files get large, we can output multiple pages.

We want to have a site (ideally static hosted on github.io) that shows:
- scenarios/metrics as rows, each model as a column (there will be fewer models than scenarios/metrics), with the ability to select rows/columns
- Clicking on each loads a single run spec and visualizes each instance (in some range), the adaptation, the prediction and all the gory details.

Mostly done by #100
Repository: stanford-crfm/helm
Issue: T5 tokenizer incorrectly instantiated with a model max length of 512 
Root cause: Configuration Error

Running the `TestT511bWindowService` test gives the following warning:

```
src/benchmark/window_services/test_t511b_window_service.py::TestT511bWindowService::test_encode
  /home/yifanmai/oss/benchmarking/venv/lib/python3.8/site-packages/transformers/models/t5/tokenization_t5_fast.py:156: FutureWarning: This tokenizer was incorrectly instantiated with a model max length of 512 which will be corrected in Transformers v5.
  For now, this behavior is kept to avoid breaking backwards compatibility when padding/encoding with `truncation is True`.
  - Be aware that you SHOULD NOT rely on t5-11b automatically truncating your input to 512 when padding/encoding.
  - If you want to encode/pad to sequences longer than 512 you can either instantiate this tokenizer with `model_max_length` or pass `max_length` when encoding/padding.
  - To avoid this warning, please instantiate this tokenizer with `model_max_length` set to your preferred value.
    warnings.warn(
```

Doesn't seem urgent, but we should take a look at some point.

This is fine because we don't rely on `model_max_length`. Also we're not using T5 at the moment.
Repository: stanford-crfm/helm
Issue: Some scenario pages are too large (>100 MB)
Root cause: Architectural Constraint

For instance, [this page](https://crfm-models.stanford.edu/static/benchmarking.html?suite=v7&runSpecs=%5B%22natural_qa%3Amode%3Dopenbook_longans%2Cmodel%3Dopenai_davinci%2Cdata_augmentation%3Dcanonical%22%5D&scenarioDisplayName=NaturalQuestions%20%2F%20mode%3A%20openbook_longans&scenarioDescription=The%20NaturalQuestions%20benchmark%20for%20question%20answering%20based%20on%20naturally-occurring%20queries%20through%20Google%20Search%20%28Kwiatkowski%20et%20al.%2C%202019%29.) downloads more than 100 MB of JSON.

![benchmarking_large_downloads](https://user-images.githubusercontent.com/185227/195714884-1594b7ba-1f03-40d0-b199-7a716d13466a.png)


Additionally, Cache-Control is incorrectly set to `no-store, must-revalidate`, meaning that the browser will always re-download the file on every page load, rather than just downloading it once.

I created a "slim" version  `scenario_state_slim.json`...doesn't like this version of the code is being used here, but let's discuss if it's a problem with the slimmed version.

Could we also turn `per_instance_stats.json` into a summarised `per_instance_stats_slim.json` as well, which just the things that the frontend uses? That file is also really big.

Does the slimming solution seem viable?  Should check this for the largest scenarios (probably language modeling or QA with long contexts).

Closing this. I looked into making `scenario_slim.json`, and there are a few ways of doing that (e.g. setting indent spaces to 0), but the savings are relatively small (maybe 10%-20%) because the bulk of the size is due to the prompt text.