zhiminy/EvalEng huggingface dataset:
Repository: pykeen/pykeen
Issue: Understanding the optimiser choice in the HPO pipeline 
GitHub labels: question

### What is your question

Hey PyKEEN team,

I'm trying to understand the optimiser choice in the HPO pipeline object. In previous PyKEEN versions, Adagrad was the default optimiser, however now it seems that from some recent runs on version 1.5 that Adam has been switched to be the default. Where in the code is this controlled? I see in the documentation that it seems Adagrad is still listed as the default if none is specified, yet it seems adam was used anyway: https://github.com/pykeen/pykeen/blob/a549a968359569c4b33c31ff382734172838cd98/src/pykeen/hpo/hpo.py#L589 

I raised, a perhaps naive and certainly not thought through, issue before on enabling an easier global view of what training setup and hyperparameter values are default across different models: https://github.com/pykeen/pykeen/issues/229 
I still think it would be cool if this could be tackled in some global way. 

### Environment

| Key             | Value                       |
|-----------------|-----------------------------|
| OS                 | posix                       |
| Platform        | Linux                       |
| Release         | 3.10.0-1160.11.1.el7.x86_64 |
| Python          | 3.8.10                      |
| PyKEEN         | 1.5.1-dev                   |
| PyKEEN Hash     | UNHASHED                    |
| PyKEEN Branch   |                             |
| PyTorch         | 1.8.1                       |
| CUDA Available? | true                        |
| CUDA Version    | 10.2                        |
| cuDNN Version   | 7605                        |

### Issue Template Checks

- [X] This is not a bug report (use a different issue template if it is)
- [X] This is not a feature request (use a different issue template if it is)

For example, if I ran the following HPO pipeline object:
``` python
 hpo_pipeline_result = hpo_pipeline(
        n_trials=10,
        dataset='Hetionet',
        model='DistMult'
)
```
I get the following resulting study.json:
```
{
  "dataset": "hetionet",
  "evaluator": "rankbased",
  "filter_validation_when_testing": false,
  "loss": "marginranking",
  "metric": "adjusted_arithmetic_mean_rank_index",
  "model": "distmult",
  "negative_sampler": "basic",
  "optimizer": "adam",
  "pykeen_git_hash": "UNHASHED",
  "pykeen_version": "1.5.1-dev",
  "regularizer": "lp",
  "training_loop": "slcwa"
}
```

also related: #598
Repository: pykeen/pykeen
Issue: 🦓 🐒  NeurIPS'21: ConE + 2 new datasets (GO21 + DDB14)
Root cause: Unimplemented Feature Gap

NeurIPS'21 brought a bunch of KGE models. This issue is for ConE. 

### Publication Link
[Modeling Heterogeneous Hierarchies with Relation-specific Hyperbolic Cones](https://papers.nips.cc/paper/2021/file/662a2e96162905620397b19c9d249781-Paper.pdf)

### Reference Implementation
Code: https://github.com/snap-stanford/ConE

### Relevance
Published at NeurIPS, probably smth people will be comparing to in future

### Additional Context
The codebase is based on RotatE, the interaction function has some hyperbolic geometry, training has an auxiliary cone normalizing component. 

The repo also contains 2 new datasets focused on hierarchies: 
* GO21 (Gene Ontology) : 89K entities, 21 relations, 796K training triples (5k val/test)
* DDB14 (DrugBank I guess) : 9K entities, 14 relations, 38K triples (4k val/test)

I refuse to add a GO or Drugbank dataset directly to pykeen. There’s no way a set of computer science authors did anything scientifically meaningful with them. 

> I refuse to add a GO or Drugbank dataset directly to pykeen. There’s no way a set of computer science authors did anything scientifically meaningful with them.

You mean more scientifically meaningful than predicting who is the coach of a football team (looking at you, `fb15k`)? :wink: 

Sure, we could probably make good arguments about all of the datasets on why they're not good. But I am an authority on biological knowledge graphs and won't let my opinion be unheard!

E.g., rather than just doing this on DrugBank and GO, why not provide an evaluation on ALL ontologies in OBO?
Repository: pykeen/pykeen
Issue: 🦓 🐒  NeurIPS'21: RotPro
GitHub labels: 💃 Model

💎 New Component

NeurIPS'21 brought a bunch of KGE models. This is for RotPro. 

### Publication Link
[Rot-Pro: Modeling Transitivity by Projection in Knowledge Graph Embedding](https://arxiv.org/pdf/2110.14450.pdf)

### Reference Implementation
Code: https://github.com/tewiSong/Rot-Pro

### Relevance
Published at NeurIPS, probably smth people will be comparing to in future

### Additional Context
The codebase is also based on RotatE, contains the interaction function and an additional regularization loss

How did you manage to subvert the new model issue template?

Oops, my bad 😬  I was attending the poster session and wrote down a few relevant KGE models. Brought back the original template!
Repository: pykeen/pykeen
Issue: 🥨🕸️ FoodWeb
Root cause: Unimplemented Feature Gap

**Describe the benchmark KG dataset**:

Which animal eats which animal? Lovely time!


**Dataset is pre-stratified into train/test/valid** Nope

**Publication(s) or website describing benchmark KG dataset**:

Hui, D. (2012). Food Web: Concept and Applications. Nature Education Knowledge 3(12):6. https://www.nature.com/scitable/knowledge/library/food-web-concept-and-applications-84077181/

**URL for downloading the benchmark KG dataset**:

https://eol.org/docs/discover/food-webs -> lead to https://zenodo.org/record/5708970/files/interactions.tsv.gz
Repository: pykeen/pykeen
Issue: Error in result tracker resolution for string
Root cause: Validation Gap

### Describe the bug

Since
```python
isinstance("string", typing.Sequence) is True
```
the following code
https://github.com/pykeen/pykeen/blob/e401e96c4400ccdc2ff7d4f7c379f16829f6fe82/src/pykeen/utils.py#L799-L810

does not convert `"string"` to `["string"]`. Thus, 
https://github.com/pykeen/pykeen/blob/e401e96c4400ccdc2ff7d4f7c379f16829f6fe82/src/pykeen/trackers/__init__.py#L63
incorrectly assumes to be provided with a list of result trackers each corresponding to the individual characters of the string.

### How to reproduce

```python
from pykeen.pipeline import pipeline

pipeline(
    dataset="nations",
    model="distmult",
    result_tracker="wandb",
)
```

produces

```python-traceback
Traceback (most recent call last):
  File "./pykeen/tmp/result_tracker_error.py", line 8, in <module>
    pipeline(
  File "./pykeen/src/pykeen/pipeline/api.py", line 992, in pipeline
    _result_tracker = resolve_result_trackers(result_tracker, result_tracker_kwargs)
  File "./pykeen/src/pykeen/trackers/__init__.py", line 78, in resolve_result_trackers
    trackers = [
  File "./pykeen/src/pykeen/trackers/__init__.py", line 79, in <listcomp>
    tracker_resolver.make(query=_result_tracker, pos_kwargs=_result_tracker_kwargs)
  File "./pykeen/venv/lib/python3.8/site-packages/class_resolver/api.py", line 255, in make
    cls: Type[X] = self.lookup(query)
  File "./pykeen/venv/lib/python3.8/site-packages/class_resolver/api.py", line 229, in lookup
    return get_cls(
  File "./pykeen/venv/lib/python3.8/site-packages/class_resolver/api.py", line 430, in get_cls
    raise ValueError(
ValueError: Invalid ResultTracker name: w. Valid choices are: ['console', 'csv', 'json', 'mlflow', 'neptune', 'python', 'tensorboard', 'wandb']
```

### Environment

| Key             | Value                    |
|-----------------|--------------------------|
| OS              | posix                    |
| Platform        | Linux                    |
| Release         | 5.4.0-91-generic         |
| Time            | Tue Dec 28 12:01:30 2021 |
| Python          | 3.8.10                   |
| PyKEEN          | 1.6.1-dev                |
| PyKEEN Hash     | e401e96c                 |
| PyKEEN Branch   | master                   |
| PyTorch         | 1.10.0+cu113             |
| CUDA Available? | true                     |
| CUDA Version    | 11.3                     |
| cuDNN Version   | 8200                     |

### Additional information

* This error was introduced with the support of multiple result trackers in https://github.com/pykeen/pykeen/pull/682.
* It can be circumvented by providing the class instead of its name.
Repository: pykeen/pykeen
Issue: Wrong Relation Mapping During Evaluation When Using Inverse Relations
Root cause: Algorithmic Error

### Describe the bug

When using inverse relations, we change the original Id of a relation from `i` to `2i` during training:
https://github.com/pykeen/pykeen/blob/e401e96c4400ccdc2ff7d4f7c379f16829f6fe82/src/pykeen/triples/triples_factory.py#L334-L347 

However, we use the original relation id during evaluation:

https://github.com/pykeen/pykeen/blob/e401e96c4400ccdc2ff7d4f7c379f16829f6fe82/src/pykeen/evaluation/evaluator.py#L487-L489

The relation ids of the validaton triples (`mapped_triples`) and `additional_filter_triples` are not adapted accordingly.

### How to reproduce

[An "unexpected" behavior can be observed when running the same setting with and without inverse relations. The loss values for both settings are very similar, but there is a large discrepancy in validation/test performance of both settings](https://wandb.ai/pykeen/inverse_relations_investigation/reports/UMLS-DistMult--VmlldzoxMzgzNjQy?accessToken=7sm584sfpfyn38e1kebaijqfw8qrg7dnflzpu19h7gpmn8iqd29gw4bv2t0h73t6):

**Without inverse relations:** 

```python3
pipeline_result = pipeline(
    dataset=UMLS,
    model=DistMult,
    dataset_kwargs=dict(
        create_inverse_triples=False,
    )
)
```

**With inverse relations:** 

```python3
pipeline_result = pipeline(
    dataset=UMLS,
    model=DistMult,
    dataset_kwargs=dict(
        create_inverse_triples=True,
    )
)
```


### Environment

| Key             | Value                         |
|-----------------|-------------------------------|
| OS              | posix                         |
| Platform        | Darwin                        |
| Release         | 19.6.0                        |
| Time            | Wed Dec 29 13:41:08 2021      |
| Python          | 3.8.12                        |
| PyKEEN          | 1.6.1-dev                     |
| PyKEEN Hash     | e401e96c                      |
| PyKEEN Branch   | fix_inverse_relations_mapping |
| PyTorch         | 1.10.0                        |
| CUDA Available? | false                         |
| CUDA Version    | N/A                           |
| cuDNN Version   | N/A                           |


### Additional information

Relevant for https://github.com/pykeen/pykeen/issues/646

Here are the relevant code parts for the `predict_h` path:
https://github.com/pykeen/pykeen/blob/e401e96c4400ccdc2ff7d4f7c379f16829f6fe82/src/pykeen/models/base.py#L312-L317

https://github.com/pykeen/pykeen/blob/e401e96c4400ccdc2ff7d4f7c379f16829f6fe82/src/pykeen/models/base.py#L524-L531

https://github.com/pykeen/pykeen/blob/e401e96c4400ccdc2ff7d4f7c379f16829f6fe82/src/pykeen/models/base.py#L487-L500

We add `+1`, but do not correctly handle the `*2` part.
Repository: pykeen/pykeen
Issue: Automate tracker options in CLI
GitHub labels: enhancement

Can we generalize this part?
We should consider that we could also pass `experiment_id`, and `experiment_name` to the MLFlow tracker.

_Originally posted by @mali-git in https://github.com/pykeen/pykeen/pull/69_

@mali-git this will make more sense after #68 is merged because we will be able to see the exciting variety
Repository: pykeen/pykeen
Issue: AttributeError: type object 'TriplesFactory' has no attribute 'from_path_binary'
Root cause: Unimplemented Feature Gap

### What is your question

I have stored the results from a TriplesFactory using 

`tf.to_path_binary('tf.pt')
`
Now I want to load it on another computer using 

`tf = TriplesFactory.from_path_binary('tf.pt')
`
but I get an error

> AttributeError: type object 'TriplesFactory' has no attribute 'from_path_binary'


How do I solve this?

### Environment

| Key             | Value                    |
|-----------------|--------------------------|
| OS              | posix                    |
| Platform        | Linux                    |
| Release         | 5.11.0-41-generic        |
| Time            | Wed Jan  5 20:34:30 2022 |
| Python          | 3.8.8                    |
| PyKEEN          | 1.6.0                    |
| PyKEEN Hash     | UNHASHED                 |
| PyKEEN Branch   |                          |
| PyTorch         | 1.9.0+cu102              |
| CUDA Available? | false                    |
| CUDA Version    | 10.2                     |
| cuDNN Version   | 7605                     |

### Issue Template Checks

- [X] This is not a bug report (use a different issue template if it is)
- [X] This is not a feature request (use a different issue template if it is)
- [X] I've read the text explaining why including environment information is important and understand if I omit this information that my issue will be dismissed

Hi, this feature has not been released yet. Either you can use the installation instructions to upgrade from 1.6.0 to the development version or sit tight for a few days - PyKEEN 1.7.0 is coming by the end of the week!
Repository: pykeen/pykeen
Issue: Add TripleRE
GitHub labels: 💃 Model

💎 New Component

### Publication Link

https://vixra.org/abs/2112.0095

### Reference Implementation

https://github.com/LongYu-360/TripleRE-Add-NodePiece

### Additional Implementations

_No response_

### Relevance

TripleRE + NodePiece currently holds the first place on the [OGB WikiKG2 leaderboard](https://ogb.stanford.edu/docs/leader_linkprop/#ogbl-wikikg2).

### Additional Context

c/o @migalkin
Repository: pykeen/pykeen
Issue: Replace custom broadcast_cat(...) by cat(broadcast_tensors(...))
GitHub labels: enhancement

good first issue

There is a function [torch.broadcast_tensors](https://pytorch.org/docs/stable/generated/torch.broadcast_tensors.html) in torch.

I think we can replace

https://github.com/pykeen/pykeen/blob/cb774cf34082bd07cd231777d5cbd2be2d8b3411/src/pykeen/utils.py#L458-L512

by 

```python
torch.cat(torch.broadcast_tensors(*tensors), dim=dim)
```

@mberr should we invite external contributors for this one?
Repository: pykeen/pykeen
Issue: allow to pass unresolved loss function to ERModel's __init__

> So then it's fine to finish this PR without adding a new model, but it would be important to write some narrative documentation explaining how one might use this interaction to construct a new model

[This](https://pykeen.readthedocs.io/en/stable/extending/models_new.html) should be the place to put it? To slightly modify the example there and create an "anonymous" model instead of declaring a new class:

```python
from pykeen.models import ERModel
from pykeen.nn import EmbeddingSpecification
model = ERModel(
    triples_factory=...,
    loss=..., # TODO: we should support passing a string here, and use the loss-resolver to resolve it
    interaction="transformer",
    entity_representations=EmbeddingSpecification(embedding_dim=64),
    relation_representations=EmbeddingSpecification(embedding_dim=64),
)
```

_Originally posted by @mberr in https://github.com/pykeen/pykeen/issues/714#issuecomment-1007712435_

Yeah, that's a great place for it!

We could also consider allowing to pass an integer as `{entity,relation}_representations` and create and internally create an `EmbeddingSpecification` with default settings from it. This might make it easier to use of `ERModel` via CLI. 🤔 

You mean like if you pass an integer it just assumes you want a vanilla embedding spec with that as the dim? Or a sequence of integers would make a sequence of embeddings? Or a sequence of tuples of ints would make fun shaped ones?

> You mean like if you pass an integer it just assumes you want a vanilla embedding spec with that as the dim? Or a sequence of integers would make a sequence of embeddings? Or a sequence of tuples of ints would make fun shaped ones?

Something like this. A shortcut which allows you to create something without having to import `EmbeddingSpecification` if you are not interested in all the possible configurations.
Repository: pykeen/pykeen
Issue: Update type annotations for kwargs

just made new release with class resolver with it https://pypi.org/project/class-resolver/0.1.1/

_Originally posted by @cthoyt in https://github.com/pykeen/pykeen/pull/718#discussion_r780516153_

`Optional[Mapping[str, Any]]` to `OptionalKwargs`

Ehh I wouldn't say we need to go back and spend a lot of effort on updating these, but it's good to know it's a shortcut for later
Repository: pykeen/pykeen
Issue: How to fit in my own dataset?
Root cause: Documentation Deficiency

### What is your question

Hi, 

I have a set of triples, like (A, dislike, B), and want to fit them into the pipeline. 

However, I'm not sure how to embed my triples, because the instructions of 'Bring Your Own Data' are all based on the existing dataset like pykeen.datasets.nations. 

Could you please teach me how to embed triples based on my own new dataset?

Thank you very much!


Best, 
Evanna 



### Environment

| Key             | Value                    |
|-----------------|--------------------------|
| OS              | posix                    |
| Platform        | Darwin                   |
| Release         | 20.6.0                   |
| Time            | Mon Jan 10 15:57:31 2022 |
| Python          | 3.8.8                    |
| PyKEEN          | 1.6.0                    |
| PyKEEN Hash     | UNHASHED                 |
| PyKEEN Branch   |                          |
| PyTorch         | 1.10.0                   |
| CUDA Available? | false                    |
| CUDA Version    | N/A                      |
| cuDNN Version   | N/A                      |

### Issue Template Checks

- [X] This is not a bug report (use a different issue template if it is)
- [X] This is not a feature request (use a different issue template if it is)
- [X] I've read the text explaining why including environment information is important and understand if I omit this information that my issue will be dismissed

Have you read this section https://pykeen.readthedocs.io/en/stable/byo/data.html#unstratified-dataset? While the example path is indeed for nations, it's up to you to provide your own path to a file. Alternatively, there are many ways to construct a [triples factory](https://pykeen.readthedocs.io/en/stable/reference/triples.html#pykeen.triples.TriplesFactory) in case you already have your triples in memory.

https://github.com/pykeen/pykeen/blob/master/notebooks/results_plots/results_plots.ipynb also gives a demo on how you can use your own triples, staring from an in-memory list of 3-tuples

Thank you for your help! 
Following your guidance, I've successfully load my own data with txt file :)


Glad it all worked out!
Repository: pykeen/pykeen
Issue: Add 5* Model
GitHub labels: 💃 Model

💎 New Component

🗑️ Missing Reference Implementation

**On which paper(s) is your requested model based.**

https://arxiv.org/abs/2006.04986

**List the original and other existing implementations of this model**

No link provided in the paper

**Relevance of this model**

This model is a logical successor of RotatE, ComplEx, etc. @mali-git can also probably the authors to help implement it in PyKEEN since they're in his group

**Ability to help and additional context**
Your background and ability to help with the implementation of the model.
Repository: pykeen/pykeen
Issue: Log loss parameters on pipeline's ResultTracker
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Hi, I am currently using **hpo_pipeline** to optimize my model. I use a **ResultTracker** (in my case wandb) in order to keep a log of my runs, and more easily compare them side by side. However one thing i noticed is that both the loss (the class) and loss_kwargs are not being logged.   

**Describe the solution you'd like**
I think if would be a good to add the loss type and loss_kwargs to the list of logged parameters, as these can have a large impact on the performance of the models.

I think it should be an easy addition. Adding something like this on the pipeline function should do the job:

```python
# Log loss parameters
_result_tracker.log_params(
    params=dict(cls=model_instance.loss.__class__.__name__, kwargs=loss_kwargs),
    prefix="loss",
)
```
I can make a PR with this change, if you would like.


Hey @Rodrigo-A-Pereira ,

thanks for your suggestion. We would highly welcome your PR!
Repository: pykeen/pykeen
Issue: Upgrade KG2E, SimplE, and TransD to new-style model

As a follow-up to  #736, these three models should be upgraded to ER models and the `pykeen.nn.emb.RepresentationModule.get_in_canonical_shape` function can be retired.

* [ ] upgrade KG2E
* [ ] upgrade SimplE
* [ ] upgrade TransD
* [ ] remove `pykeen.nn.emb.RepresentationModule.get_in_canonical_shape`

#### Dependencies
* [x] #736
Repository: pykeen/pykeen
Issue: Could you publish some experimental results ？
GitHub labels: enhancement

I got the result of TransE on WN18RR dataset as follows,  with 100 epoch and default parameter settings.

mean_rank={'best': 7396.350889192887, 'worst': 7396.3558481532145, 'avg': 7396.353368673051}, mean_reciprocal_rank={'best': 0.11426725607718224, 'worst': 0.11426725590337292, 'avg': 0.1142672559902559}, hits_at_k={'best': {1: 0.0032489740082079343, 3: 0.19493844049247605, 5: 0.24811901504787962, 10: 0.2973666210670315}, 'worst': {1: 0.0032489740082079343, 3: 0.19493844049247605, 5: 0.24811901504787962, 10: 0.2973666210670315}, 'avg': {1: 0.0032489740082079343, 3: 0.19493844049247605, 5: 0.24811901504787962, 10: 0.2973666210670315}}, adjusted_mean_rank=0.36492311337297334.

it seems a bit lower than the result reported in the paper


Hi @Woooooody,
we have made all our experimental artifacts available at https://github.com/pykeen/benchmarking.
The best TransE configuration that we found for WN18RR is available at https://github.com/pykeen/benchmarking/tree/master/ablation/results/transe/wn18rr/random/adam/2020-05-20-03-15_00b22931-c88f-4d90-a606-c94a31a80516/0003_wn18rr_transe/best_pipeline.
Please note that we re-named the OWA training loop to SLCWA. The configuration files contain the previous nomenclature.
Repository: pykeen/pykeen
Issue: Encapsulate inverse relations inside Model
Root cause: Architectural Constraint

**Is your feature request related to a problem? Please describe.**
The inverse triples usage is currently somewhat inconsistent, e.g., we expected the training instances to contain inverse triples, but do require evaluation triples to *not* contain them. 

**Describe the solution you'd like**
Encapsulate inverse triples logic into `Model` class. This also would be coherent with our description in the [resource](https://arxiv.org/abs/2007.14175) and [benchmarking](https://arxiv.org/abs/2006.13365) paper, where we explain why using inverse relations is a *modelling* decision rather than a simple data augmentation technique.