zhiminy/EvalEng huggingface dataset:
Repository: pykeen/pykeen
Issue: How to use the trained model to predict ？
Root cause: Documentation Deficiency

After I train the model on my own dataset,  I got a file named  `trained_model.pkl` ,how to load the saved model and use it to predict?


Hi,
you can load a trained model as follows (https://pytorch.org/tutorials/beginner/saving_loading_models.html):
```python
import torch
path = 'path/to/trained_model.pkl'
model = torch.load(path)
model.eval()
```

[Here](https://github.com/pykeen/pykeen/blob/master/notebooks/hello_world/Hello%20World!.ipynb) we provide a tutorial of how to introspect a trained model and how to make predictions.

We provide further functions to make predictions, e.g., `predict_scores()`, `predict_scores_all_tails()`, and `predict_scores_all_heads()` which are defined in the [base](https://github.com/pykeen/pykeen/blob/master/src/pykeen/models/base.py) model class. However, these functions require that the entities and relations are already mapped to their unique ids (https://pykeen.readthedocs.io/en/latest/tutorial/performance.html#entity-and-relation-ids). 
Entities and relations in triples can be mapped to their unique ids as follows:

```python
import numpy as np
# Here, we assume that we trained the model on the 'Nations' dataset
triples = np.array([['brazil','conferences','china']])
mapped_triples = model.triples_factory.map_triples_to_id(triples=triples)
```

You then can, for instance, use the function `predict_scores()` to make predictions:

```python
model.predict_scores(triples=mapped_triples)
```

If you want to map entities/relations individually, you can make use of the internal mappings `model.triples_factory.entity_to_id` and `model.triples_factory.relation_to_id`.

A further example of how to make predictions can be found [here](https://pykeen.readthedocs.io/en/latest/tutorial/making_predictions.html).

> A further example of how to make predictions can be found [here](https://pykeen.readthedocs.io/en/latest/tutorial/making_predictions.html).

I would definitely suggest using the models from the tutorial from Mehdi's second post. If you're very comfortable with PyTorch, you might consider going into the guts he described in his first post.

Thank you so much, great work ：）
Repository: pykeen/pykeen
Issue: NodePiece representations with anchor nodes
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
The current implementation of NodePiece representations only provides the relation tokenization, but not the variant with anchor nodes. E.g., for #715 , we need the anchor node variant.

**Describe the solution you'd like**
Implement the anchor node variant as part of the existing `NodePiece` representations. The current implementation can provide some guidance.

**Additional context**
* Official Code: https://github.com/migalkin/NodePiece
* Paper: https://arxiv.org/abs/2106.12144


Some external library will definitely be needed here (please no networkx 😅 ) as the speed is crucial for larger graphs. 
The core procedure for anchor mining is BFS, so some scalable parallel way would be most preferable. 

* With `igraph`(original implementation) datasets up to YAGO 3-10 (123k nodes) can be tokenized relatively fast (5 min on a laptop for yago, 1 min for Fb15k-237), but igraph is a single-core software, so it won't work well on larger graphs. For OGB WikiKG2 (2.5M nodes) it took ~30-ish hours. Possible solution - partition the graph with METIS (built into PyG for example) and run anchor mining in parallel for those partitions. 
* `graph-tool` and `networkit` seem to be much faster than `igraph` in subgraph extraction, but I didn't experiment with them since they often have tricky C/C++ dependencies with which I had troubles compiling for an M1 mac. 
* PyG has a built-in [k-hop subgraph](https://pytorch-geometric.readthedocs.io/en/latest/_modules/torch_geometric/utils/subgraph.html#k_hop_subgraph) and can do this even on CUDA, but I'm not sure in memory/parallelism 

For a limited functionality, e.g., anchor node selection by degree only, https://docs.scipy.org/doc/scipy/reference/sparse.csgraph.html may also be a viable alternative.
Repository: pykeen/pykeen
Issue: NodePiece with joint relation and anchor tokenization
Root cause: Unimplemented Feature Gap

> The `NodePieceRepresentation` module works fine, but the `NodePiece` model doesn't work as of now as it needs fixing the instantiation procedure according to [#755 (comment)](https://github.com/pykeen/pykeen/pull/755#discussion_r793911213)

Relation *or* anchor tokenization should work now (but not both at once :sweat_smile: )

_Originally posted by @mberr in https://github.com/pykeen/pykeen/issues/755#issuecomment-1025571144_
Repository: pykeen/pykeen
Issue: MixtureAnchorSelection contains duplicates
Root cause: Algorithmic Error

I found a problem in the `MixtureAnchorSelection`:

https://github.com/pykeen/pykeen/blob/78013e1f2a0fa43e49550422a77b0683d4653902/src/pykeen/nn/node_piece.py#L273-L274

Those lines do not guarantee unique anchors, that is, if a node is in the top-K degree and simultaneously is in the top-k PageRank, it would be counted twice as an anchor, and that is an error - anchors have to be unique nodes. 

This can be reproduced on FB15k-237 asking for 100 anchors of 50% top-degree and 50% top PPR:
```python
    dataset = FB15k237(create_inverse_triples=True)
    loss = NSSALoss(margin=15)
    model = NodePiece(
        triples_factory=dataset.training,
        loss=loss,
        random_seed=42,
        num_tokens=3,
        aggregation="mlp",
        tokenizers='anchor',
        tokenizers_kwargs=[
            dict(
                selection="MixtureAnchorSelection",
                searcher="ScipySparseAnchorSearcher",
                selection_kwargs=dict(
                    selections=['degree', 'pagerank'],
                    ratios=[0.5, 0.5],
                    num_anchors=100,
                ),
            ),
            dict()
        ]
    )
```

This returns 91 unique anchor (after doing `np.unique(anchors)`). 
I suppose this is a separate issue to open?

_Originally posted by @migalkin in https://github.com/pykeen/pykeen/issues/770#issuecomment-1029600824_
Repository: pykeen/pykeen
Issue: Is the data being updated
GitHub labels: question

### What is your question

I would like to know if the datasets that could be loaded through PyKEEN is constantly being updated

Thanks

### Environment

python

### Issue Template Checks

- [X] This is not a bug report (use a different issue template if it is)
- [X] This is not a feature request (use a different issue template if it is)
- [X] I've read the text explaining why including environment information is important and understand if I omit this information that my issue will be dismissed

I am not sure whether I get your question correctly. 

Are you wondering whether new datasets get added to pykeen from time to time? If yes, yes, You can look at the history of already added and pending datasets [here](https://github.com/pykeen/pykeen/issues?q=label%3A%22%F0%9F%92%BE+Dataset%22+is%3Aclosed) by filtering for the "💾 Dataset" label.

If you were wondering whether the already included datasets change: no, typically not. We do not host most of them ourselves but try to use persistent web sources, such as https://zenodo.org/

My question was more of the latter one, 
in particular, I am interested in the clinical knowledge graph (aka CKG) dataset. 
ever since its first release, there has been two updates, thus [the latest version of the dataset is 3.0](https://data.mendeley.com/datasets/mrcf7f4tc2/1) .
Have you guys managed to update CKG datasets to version 3.0, or should I load it manually? 
Thanks

CKG has been added by @cthoyt in https://github.com/pykeen/pykeen/pull/213 and merged to `master` on 13 Dec 2020. afaik, there was no further update (in PyKEEN) since then.

This is the source URL
https://github.com/pykeen/pykeen/blob/9cf6da4e1ea9430f48e4ef42366effb0b0fa7c86/src/pykeen/datasets/ckg.py#L23

EDIT: according to https://github.com/MannLabs/CKG/issues/42, this is version 2.

I remember the whole experience of CKG to be frustrating in several ways:

1. the technical means through which the data was distributed
2. the huge mismatch between what was described in their pre-print (recently accepted for publication iirc) and what was actually made available
3. the opaqueness in the KG generation due to the difficulty of navigating their code

Maybe at minimum we should do a better job annotating the versions of data (there's also an old issue for updating OGB in #583, but I also am still upset the way they quickly dismiss feedback and questions on their GitHub, so not sure how much I want to engage with that again either)

I might get my hands dirty with their code, then. 
Thanks for the answer!
Repository: pykeen/pykeen
Issue: Add OpenEA Datasets
Root cause: Unimplemented Feature Gap

**Describe the benchmark KG dataset**: These are benchmark datasets for the task of entity alignment, i.e. matching the same entities across different sources. The datasets contain samples from DBpedia, Wikidata and Yago with varying sizes, connectivity. Furthermore in some cases they contain a cross-lingual setting.

**Dataset is pre-stratified into train/test/valid**: Only for the entity alignment ins a pre-stratification available for 5-folds.

**Publication(s) or website describing benchmark KG dataset**: [Github](https://github.com/nju-websoft/OpenEA#dataset-overview) [Paper](http://www.vldb.org/pvldb/vol13/p2326-sun.pdf)

**URL for downloading the benchmark KG dataset**: [Dropbox Link](https://www.dropbox.com/s/xfehqm4pcd9yw0v/OpenEA_dataset_v2.0.zip?dl=0)

I would volunteer to integrate these datasets following a suggestion from @cthoyt .
Since I saw that @mberr already added some entity alignment datasets in #403 I would follow the general design decisions from there.


@dobraczka in addition to #403 , it might be interesting to retain some of the alignment information, e.g., by allowing to have the "joint" graph, and a special alignment relation. This may also be part of a future, separate PR though.

This is a good idea. But might be better suited for a future PR, where the datasets from #403 could be adapted as well? If it's fine by you I would then start working on the PR to integrate the datasets.

@dobraczka in the mean time, do you think you could petition the authors of this dataset to put it in a more stable location (e.g., Zenodo or FigShare)? Datasets on dropbox can disappear at anytime at the author's (or their institution's) discretion.
Repository: pykeen/pykeen
Issue: Fix usage of nonzero for PyTorch 1.7+
Root cause: External Dependency Breakage

**Is your feature request related to a problem? Please describe.**
The usage of `Tensor.nonzero()` (without arguments) has been deprecated, cf. https://github.com/pytorch/pytorch/pull/40187

At least in 

https://github.com/pykeen/pykeen/blob/630d8721fdb3a830a25be01a531bac436787fd96/src/pykeen/evaluation/evaluator.py#L360

we use the now deprecated version.

**Describe the solution you'd like**
Change all usages of `nonzero` to use to future-proof variant, i.e. in the above mentioned case

```python
filter_batch = (entity_filter_test & relation_filter).nonzero(as_tuple=False) 
```

**Describe alternatives you've considered**
Keep it as it is. This might lead to problems once PyTorch removes the deprecated behaviour.

**Additional context**
Warning found in this notebook: https://gist.github.com/cthoyt/190233fd98a11306ceb13f2ee0e95a9e (scroll *all* the way to the bottom)
Repository: pykeen/pykeen
Issue: ResultTracker tracks only hits_at_k
Root cause: Algorithmic Error

### Describe the bug

I am running a training script from the inductive LP tutorial with the wandb tracker and see only hits_at_k metrics.
Tracing the logging in the debugger it seems that `_iter_rows()` is not working properly:

https://github.com/pykeen/pykeen/blob/c39713588cb21bb8a0ab0f695fd405edf514fe75/src/pykeen/evaluation/rank_based_evaluator.py#L257-L265

Particularly, this condition is never met and nothing gets yielded:
https://github.com/pykeen/pykeen/blob/c39713588cb21bb8a0ab0f695fd405edf514fe75/src/pykeen/evaluation/rank_based_evaluator.py#L264-L265

Might it be due to the recent changes in evaluation metrics organization?

### How to reproduce

The script at the end of the tutorial https://pykeen.readthedocs.io/en/latest/tutorial/inductive_lp.html with the `WANDBResultTracker`

### Environment

| Key             | Value                    |
|-----------------|--------------------------|
| OS              | posix                    |
| Platform        | Darwin                   |
| Release         | 21.2.0                   |
| Time            | Thu Feb 17 21:45:24 2022 |
| Python          | 3.8.5                    |
| PyKEEN          | 1.7.1-dev                |
| PyKEEN Hash     | c3971358                 |
| PyKEEN Branch   | master                   |
| PyTorch         | 1.10.0                   |
| CUDA Available? | false                    |
| CUDA Version    | N/A                      |
| cuDNN Version   | N/A                      |

### Additional information

_No response_

MWE
```python
from pykeen.pipeline import pipeline

pipeline(
    dataset="nations",
    model="distmult",
    result_tracker="wandb",
    result_tracker_kwargs=dict(project="issue_791"),
    stopper="early",
    stopper_kwargs=dict(frequency=1, patience=1000),
)
```
Repository: pykeen/pykeen
Issue: `get_preferred_device` always returns CPU
Root cause: Documentation Deficiency

### Describe the bug

It looks like that by default any created model is placed on a CPU. 
This leads to the error when the following code always returns CPU since all model parameters are on a CPU

https://github.com/pykeen/pykeen/blob/09f8b49637bab55c14febb16f29eebe81eeedb73/src/pykeen/models/base.py#L156-L173

It never reaches line 170 resolving any CUDA devices

### How to reproduce

Running an inductive LP tutorial model on a GPU

### Environment

| Key             | Value                    |
|-----------------|--------------------------|
| OS              | posix                    |
| Platform        | Darwin                   |
| Release         | 21.2.0                   |
| Time            | Fri Feb 18 16:17:24 2022 |
| Python          | 3.8.5                    |
| PyKEEN          | 1.7.1-dev                |
| PyKEEN Hash     | 09f8b496                 |
| PyKEEN Branch   | master                   |
| PyTorch         | 1.10.0                   |
| CUDA Available? | false                    |
| CUDA Version    | N/A                      |
| cuDNN Version   | N/A                      |

### Additional information

So far the only way to solve this is to explicitly send the model to `device` after its creation:

```python
model = AnyModel(...).to(resolve_device())
```

Hm, I think this is intended behaviour: if nothing is provided, the model resides on `cpu`.

It was introduced with https://github.com/pykeen/pykeen/pull/730 and fixes a few issues of reloading a model for CPU inference (https://github.com/pykeen/pykeen/issues/748 https://github.com/pykeen/pykeen/issues/727).

EDIT: the pipeline does handle this for you. 
https://github.com/pykeen/pykeen/blob/09f8b49637bab55c14febb16f29eebe81eeedb73/src/pykeen/pipeline/api.py#L1082
However, if you do not use it, but want to use the components separately, you need to move the model to the device of your choice.

I see, yes the script has separate components, so I'll consider this an expected behavior. 

Closing the issue then!
Repository: pykeen/pykeen
Issue: Add PharmKG-8k Dataset
Root cause: Unimplemented Feature Gap

**Describe the benchmark KG dataset**:

The dataset is a small biomedical KG comprising genes, compounds and diseases.  

**Dataset is pre-stratified into train/test/valid** [x ]

**Publication(s) or website describing benchmark KG dataset**: https://doi.org/10.1093/bib/bbaa344

**URL for downloading the benchmark KG dataset**: https://github.com/biomed-AI/PharmKG/tree/master/data/PharmKG-8k
Repository: pykeen/pykeen
Issue: New split algorithm for constrained evaluation
Root cause: Unimplemented Feature Gap

> What about adding a mode that moves all entities that won't get used in evaluation over to training?

I suppose you mean the triples containing entities which are not in `evaluation_entity_whitelist`? We could think about this, but it would change the split ratio of the subsets' number of triples. 

Maybe we add this functionality in our splitting functionalities (and emit appropriate warnings)? Or we look at it from a different perspective: we first split all triples of interest by `split_ratio`. Then we extend the training set by additional triples. Practically we need to make sure that the ID-mapping stays consistent.

_Originally posted by @mberr in https://github.com/pykeen/pykeen/pull/62#issuecomment-679878694_
Repository: pykeen/pykeen
Issue: Bootstrapping Ranks for Confidence Estimates
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
While we can get variances in (rank-based) metrics by repeated training of a model, there is a cheaper alternative which only needs a single trained model.

**Describe the solution you'd like**
The rank-based evaluator stores the ranks for each triple. In the finalization stage, it then computes several aggregation metrics. We could use [bootstrapping](https://en.wikipedia.org/wiki/Bootstrapping_(statistics)) to extract confidence estimates for the computes metrics.
Repository: pykeen/pykeen
Issue: Add PharmKGFull
Root cause: Unimplemented Feature Gap

**Describe the benchmark KG dataset**:

The dataset is a small biomedical KG comprising genes, compounds and diseases.  

**Dataset is pre-stratified into train/test/valid** []

**Publication(s) or website describing benchmark KG dataset**: https://doi.org/10.1093/bib/bbaa344

**URL for downloading the benchmark KG dataset**: https://zenodo.org/record/4077338
Repository: pykeen/pykeen
Issue: DCG and Normalized DCG
Root cause: Unimplemented Feature Gap

Hi @mberr / @mali-git

I couldn't find the implementation of DCG and NDCG in PyKeen. That is a nice measure to have as many research papers report their results based on that.

NDCG Wiki: https://en.wikipedia.org/wiki/Discounted_cumulative_gain#Normalized_DCG
NDCG Scikit: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.ndcg_score.html

Thanks,

Could you give some examples of papers using knowledge graph embedding models that report one or both of these?

Hi @cthoyt 

Here there are some examples:

- https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7273238/
- https://arxiv.org/pdf/2006.07331.pdf
- https://dl.acm.org/doi/10.1145/3460231.3474272
- https://enricopal.github.io/publications/Translational_Models_for_Item_Recommendation.pdf

Cheers,

I'm not super convinced by this list:

- https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7273238/ - I'm going to ignore this article because it seems to be done in a very uninformed way (in 2020 it's not acceptable to compare your models against TransE and family, even less to apply it to a biomedical knowledge graph)
- https://arxiv.org/pdf/2006.07331.pdf - I'm not convinced that DCG is a good metric in this case since it's built on estimations of false negatives
- https://dl.acm.org/doi/10.1145/3460231.3474272 - Not open access, can't assess
- https://enricopal.github.io/publications/Translational_Models_for_Item_Recommendation.pdf - These metrics are used to analyze a ranking function built on top of the knowledge graph embeddings (PyKEEN only implements the link prediction task)

Could you please write a bit more about why we might be motivated to include this metric in PyKEEN?

That being said, we could import the implementations from https://github.com/AstraZeneca/rexmex/blob/44f453ff20e92569270b9e1cfcb75b44b7839128/rexmex/metrics/ranking.py#L425-L451, which seem to be formulated the same way as classification metrics (though categorized as ranking metrics?)

Hey,

Sorry for my late reply. Assume that we want to use link prediction for a recommendation task. Although I agree that nDGC may not be often used in other KG-based tasks, it is a well-known measure in the recommendation world. It is also a ranking measure. You may like to have a look at this: https://arxiv.org/abs/1304.6480

I will implement it myself and share the code with you later.

Thanks,

I share a function for calculating the binary version of nDCG:

```python

from sklearn.metrics import ndcg_score

def cal_ndcg(true_value, pred_values):
    
    """
    :param true_value: this is the true value, e.g., the true tail
    :param pred_values: these are the prediction values, e.g., the predicated tails
    :return: nDCG Score
    """
    
    k = len(pred_values) # it is also possible to pass in `k` as an argument
    true_relevance = [1] + [0] * (k - 1)
    
    scores = [0] * k
    scores[pred_values.index(true_value)] = 1
    
    ndcg = ndcg_score([true_relevance], [scores])

    return ndcg
```

I close this issue.
Repository: pykeen/pykeen
Issue: Report marginal evaluation metrics
Root cause: Unimplemented Feature Gap

In the opposite scenario to #62/#83, we want to report several different sets of metrics based on each relation type. For example, for hetionet, you might want to specify edges or groups of edges to evaluate together:

```python
from pykeen.pipeline import pipeline

pipeline_results = pipeline(
    dataset='Hetionet',
    model='RotatE',
    evaluation_relation_whitelist=[
         {'CtD', 'CpD'},  # evaluate the compund treats disease and compounds palliates disease relations together
        'CcSE',  # evaluate the compound causes side effect alone
        {'CdG', 'CuG', 'CbG'},  # compound/gene regulations/bindings
        {'GiG', 'GcG',  'Gr>G'},  # gene/gene relations,
        None,  # special entry None should group all remaining that haven't appeared in the rest
    ],
)
```

- It should theoretically possible to have duplicates appearing throughout the subsets.
- `None` should be a special entry that groups everything you haven't considered. Maybe raise error if all have been covered and None is also added?
- `all` (or something else that can be written in a JSON configuration) should be a special entry that will do everything. Alternatively, "all" could always be reported

Alternate idea, using a dict for named subsets

```python
from pykeen.pipeline import pipeline

pipeline_results = pipeline(
    dataset='Hetionet',
    model='RotatE',
    evaluation_relation_whitelist={
        'Compound-Disease': {'CtD', 'CpD'},
        'Compound-SideEffect': 'CcSE',
        'Compound-Gene': {'CdG', 'CuG', 'CbG'},
        'Gene-Gene': {'GiG', 'GcG',  'Gr>G'},
        'Remainder': None,
    },
)
```

It might be better overall to force usage of dictionaries so we can switch on the type (mapping vs. collection)

I just read through the pipeline and it seems like this is going to be quite a pain in the butt to do this...