zhiminy/EvalEng huggingface dataset:
Repository: pykeen/pykeen
Issue: Can't save the model constructed by 'make_model_cls'
Root cause: Architectural Constraint

### Describe the bug

When I try to save the model constructed by 'make_model_cls', I get an error:
`AttributeError: Can't pickle local object 'make_model_cls.<locals>.ChildERModel'`



### How to reproduce

```
from pykeen.pipeline import pipeline
from pykeen.nn.modules import TransEInteraction
from pykeen.models import make_model_cls

model = make_model_cls(
    interaction=TransEInteraction,
    interaction_kwargs={'p': 2},
    dimensions={'d': 100},
)
results = pipeline(
    dataset='Nations',
    model=model,
    training_kwargs= {
        "batch_size": 512,
        "num_epochs": 1
        },
)

results.save_to_directory('save')
```

### Environment

| Key             | Value                    |
|-----------------|--------------------------|
| OS              | posix                    |
| Platform        | Linux                    |
| Release         | 5.15.0-50-generic        |
| Time            | Tue Oct 18 11:56:01 2022 |
| Python          | 3.9.12                   |
| PyKEEN          | 1.9.1-dev                |
| PyKEEN Hash     | UNHASHED                 |
| PyKEEN Branch   |                          |
| PyTorch         | 1.12.0+cu116             |
| CUDA Available? | true                     |
| CUDA Version    | 11.6                     |
| cuDNN Version   | 8400                     |


### Additional information

_No response_

### Issue Template Checks

- [X] This is not a feature request (use a different issue template if it is)
- [X] This is not a question (use the [discussions](https://github.com/pykeen/pykeen/discussions/new) forum instead)
- [X] I've read the text explaining why including environment information is important and understand if I omit this information that my issue will be dismissed
Repository: pykeen/pykeen
Issue: save_to_directory partially working
Root cause: Environment Incompatibility

Hi everyone, 

i'm having issues with the function save_to_directory.
In particular, given the folder i pass as argument, it is able to generate the files

```
metadata.json
results.json
trained_model.pkl
```

but does not generate the folder `training_triples`.

This is the source code (i used nation dataset to test it)
I'm currently under Windows with Python 3.9.7

```python
test_path = NATIONS_TEST_PATH

emb_training = TriplesFactory.from_path(
    train_path,
    create_inverse_triples=True,
)

emb_testing = TriplesFactory.from_path(
    test_path,
    entity_to_id=emb_training.entity_to_id,
    relation_to_id=emb_training.relation_to_id,
    create_inverse_triples=True,
)

result = pipeline(
    training=emb_training,
    testing=emb_testing,
    model=emb_model,
    model_kwargs=dict(embedding_dim=emb_dim),
    epochs=emb_epochs, 
)

# save mappings
result.save_to_directory('dest')
```

Hi @giuspillo ,

would you mind re-opening the ticket with the "bug report" template and fill-in the environment information (e.g., which PyKEEN version)?

With the current `master` branch, I cannot reproduce your observation:
```python
from pykeen.pipeline import pipeline
from pykeen.triples import TriplesFactory
from pykeen.datasets.nations import NATIONS_TEST_PATH, NATIONS_TRAIN_PATH

emb_training = TriplesFactory.from_path(
    NATIONS_TRAIN_PATH,
    create_inverse_triples=True,
)

emb_testing = TriplesFactory.from_path(
    NATIONS_TEST_PATH,
    entity_to_id=emb_training.entity_to_id,
    relation_to_id=emb_training.relation_to_id,
    create_inverse_triples=True,
)

result = pipeline(
    training=emb_training,
    testing=emb_testing,
    model="TransE",
    epochs=0,
)
# save mappings
result.save_to_directory("/tmp/test")
```

```console
$ ls /tmp/test
metadata.json  results.json  trained_model.pkl  training_triples
```
Repository: pykeen/pykeen
Issue: Recommend best reproducing settings and parameters to reproduce ComplEx+N3
Root cause: Documentation Deficiency

### Problem Statement

I tried to reproduce the best ComplEx + N3 scores with dataset FB15K237.
The settings are as below:
pipeline_result = pipeline(
        dataset=ds.FB15k237(),
        model=ComplEx,
        model_kwargs=dict(embedding_dim=512, entity_initializer="xavier_uniform", relation_initializer="xavier_uniform"),
        loss=CrossEntropyLoss,
        loss_kwargs={"reduction": "mean"},
        regularizer=LpRegularizer,
        regularizer_kwargs=dict(weight=5e-2, p=3.0, ),
        optimizer="adagrad",
        optimizer_kwargs=dict(lr=0.5),
        evaluator=RankBasedEvaluator,
        training_loop="SLCWA",
        negative_sampler="basic",
        negative_sampler_kwargs={"num_negs_per_pos": 10},
        training_kwargs={
            "num_epochs": 1000,
            "batch_size": 1024
        },
The scores are much less than the reported scores in papers.  I tried the parameters from this paper "You can teach an Old Dog New Tricks! On Training Knowledge Graph Embeddings" and this repo https://github.com/facebookresearch/kbc, the scores are even lower.
I feel it is pretty hard to find the best parameters. Is it Ok that Pykeen can provide settings/parameters for FB15K237 to reproduce scores?
Especially for ComplEx and Canonical Tensor Decomposition.

### Describe the solution you'd like

Is it Ok that Pykeen can provide settings/parameters or checkpoints for datasets to reproduce scores in papers?

### Describe alternatives you've considered

Is it Ok that Pykeen can provide settings/parameters or checkpoints for FB15K237 to reproduce scores in paper?
Especially for ComplEx and Canonical Tensor Decomposition.

### Additional information

_No response_

### Issue Template Checks

- [X] This is not a bug report (use a different issue template if it is)
- [X] This is not a question (use the [discussions](https://github.com/pykeen/pykeen/discussions/new) forum instead)

Hi,

reproducing KGE papers is unfortunately sometimes not easy. We curate some experiment configurations at [`src/pykeen/experiments`](https://github.com/pykeen/pykeen/tree/master/src/pykeen/experiments), which you can run from console as described in https://github.com/pykeen/pykeen#reproduction 

In addition, we make all configurations from [our reproducibility and benchmarking study](https://arxiv.org/abs/2006.13365) available at https://github.com/pykeen/benchmarking. Please note that these experiments were run with an older version of PyKEEN, and you may need to adjust some parameter names to account for changes made since then.
Repository: pykeen/pykeen
Issue: Can't run RGCN with negative_sampler="basic"
Root cause: Resource Mishandling

### Describe the bug

when I run RGCN model and set negative_sampler="basic", I get an error:
RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.

### How to reproduce
```python
from pykeen.pipeline import pipeline

train = ...  # todo: missing
test = ...  # todo: missing
result = pipeline(
    model="RGCN",
    training=train,
    testing=test,
    device="gpu",
    model_kwargs=dict(embedding_dim=50),
    training_kwargs=dict(num_epochs=100, batch_size=20480),
    negative_sampler="basic",
    negative_sampler_kwargs=dict(num_negs_per_pos=10),
    training_loop="slcwa",
    optimizer="Adam",
    optimizer_kwargs=dict(lr=1e-3),
)
```
### Environment

| Key             | Value                    |
|-----------------|--------------------------|
| OS              | posix                    |
| Platform        | Linux                    |
| Release         | 5.15.65+                 |
| Time            | Tue Nov  8 02:31:02 2022 |
| Python          | 3.7.12                   |
| PyKEEN          | 1.8.1                    |
| PyKEEN Hash     | UNHASHED                 |
| PyKEEN Branch   |                          |
| PyTorch         | 1.11.0                   |
| CUDA Available? | true                     |
| CUDA Version    | 11.0                     |
| cuDNN Version   | 8005                     |

### Additional information

_No response_

### Issue Template Checks

- [X] This is not a feature request (use a different issue template if it is)
- [X] This is not a question (use the [discussions](https://github.com/pykeen/pykeen/discussions/new) forum instead)
- [X] I've read the text explaining why including environment information is important and understand if I omit this information that my issue will be dismissed

When I comment out this line, it will work normally:
        negative_sampler="basic"


Hi @yj1227 ,

could you please try again with the current PyKEEN version (1.9)?

Also, it would be great to have a self-contained reproduction example (in your example, `train` and `test` are unknown), as well as trying to reduce the example to a bare minimum, e.g.,
```python
from pykeen.pipeline import pipeline

result = pipeline(
    model="RGCN",
    dataset="nations",
    model_kwargs=dict(embedding_dim=2),
    training_kwargs=dict(num_epochs=1),
    negative_sampler="basic",
    negative_sampler_kwargs=dict(num_negs_per_pos=10),
    training_loop="slcwa",
)
```
Repository: pykeen/pykeen
Issue: Some examples in the docs are out of date
Root cause: Documentation Deficiency

### Problem Statement

Many examples in the readthedocs of this project are wrong, can you please correct them? For a beginner, it is difficult to get started, so readthedocs can write in detail?

### Describe the solution you'd like

Many examples in the readthedocs of this project are wrong, can you please correct them? For a beginner, it is difficult to get started, so readthedocs can write in detail?

### Describe alternatives you've considered

Many examples in the readthedocs of this project are wrong, can you please correct them? For a beginner, it is difficult to get started, so readthedocs can write in detail?

### Additional information

Many examples in the readthedocs of this project are wrong, can you please correct them? For a beginner, it is difficult to get started, so readthedocs can write in detail?

### Issue Template Checks

- [X] This is not a bug report (use a different issue template if it is)
- [X] This is not a question (use the [discussions](https://github.com/pykeen/pykeen/discussions/new) forum instead)

Could you please link the examples that do not work?

Hi @johnoooooo, thanks for the feedback but we can't make any updates without a bit more specificity. I'm going to close this for now, feel free to comment if you can provide a specific example for us to update.
Repository: pykeen/pykeen
Issue: Unable to print evaluation results
Root cause: Documentation Deficiency

### Describe the bug

When following the [documentation](https://pykeen.readthedocs.io/en/stable/tutorial/first_steps.html#training-a-model) to print evaluation results:


`results = evaluator.evaluate(model=model,
    mapped_triples=mapped_triples,
    batch_size=1024,
    additional_filter_triples=[
        dataset.training.mapped_triples,
        dataset.validation.mapped_triples,
    ],)`
`print(results)`

I get the following message (both in Spyder 5 and Jupyter Notebook):

*<pykeen.evaluation.rank_based_evaluator.RankBasedMetricResults object at 0x0000015D54B0DF10>*

Can you please assist on how to print the corresponding evaluation results?

In section below I'm showing the full script I'm using.

Thanks!

### How to reproduce

![image](https://user-images.githubusercontent.com/40613836/204057676-172b92f9-b011-4e3e-a32f-b97996c500bb.png)


### Environment

pykeen version 1.8.1

### Additional information

I'm using Spyder 5 and Jupyter Notebook in Windows 10 environment

### Issue Template Checks

- [X] This is not a feature request (use a different issue template if it is)
- [X] This is not a question (use the [discussions](https://github.com/pykeen/pykeen/discussions/new) forum instead)
- [X] I've read the text explaining why including environment information is important and understand if I omit this information that my issue will be dismissed

Hi @dlopezyse ,

the result of the [evaluate](https://pykeen.readthedocs.io/en/stable/api/pykeen.evaluation.evaluate.html) method is an object of type [`MetricResults`](https://pykeen.readthedocs.io/en/stable/api/pykeen.evaluation.MetricResults.html); in your specific case a [`RankBasedMetricResults`](https://pykeen.readthedocs.io/en/stable/api/pykeen.evaluation.RankBasedMetricResults.html). They offer a few methods to export the results either as dictionary, or as a pandas dataframe. Please take a look at the linked pages to find a method which fits your needs.

P.S.: For future issues, it would be better to include the code as text rather than an image; even better is to additionally add code syntax highlighting as explained [here](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks).

Thanks a lot @mberr  
I'll follow your guidance 👍🏻
Repository: pykeen/pykeen
Issue: Where did `pykeen.models.predict` go in PyKEEN > 1.9.0?
Root cause: Documentation Deficiency

### Describe the bug

Hello,

while trying to import the following `from pykeen.models.predict import get_relation_prediction_df`. I get the error that:  `No module named 'pykeen.models.predict'`.
I am using google colab , and I installed pykeen using `! pip install git+https://github.com/pykeen/pykeen.git`.

could you figure out why the "predict" is not under "models" in my case ?

thank you

### How to reproduce

On a google Colab notebook add the following:
! pip install git+https://github.com/pykeen/pykeen.git
from pykeen.models.predict import get_relation_prediction_df


### Environment

<html>
<body>
<!--StartFragment-->



Running `pykeen.env()` output : 
OS | posix
-- | --
Platform | Linux
Release | 5.10.133+
Time | Thu Dec 8 14:27:20 2022
Python | 3.8.15
PyKEEN | 1.9.1-dev
PyKEEN Hash | UNHASHED
PyKEEN Branch |  
PyTorch | 1.13.0+cu116
CUDA Available? | true
CUDA Version | 11.6
cuDNN Version | 8302

<!--EndFragment-->
</body>
</html>

### Additional information

_No response_

### Issue Template Checks

- [X] This is not a feature request (use a different issue template if it is)
- [X] This is not a question (use the [discussions](https://github.com/pykeen/pykeen/discussions/new) forum instead)
- [X] I've read the text explaining why including environment information is important and understand if I omit this information that my issue will be dismissed

Note that you're on an unreleased version of PyKEEN and we have refactored/moved this module. I'd suggest sticking to 1.9.0 for now or reading the migration guide on the documentation for the development version of PyKEEN at https://pykeen.readthedocs.io/en/latest/reference/predict.html#migration-guide. Please let us know if there's something missing from this guide, and we will make appropriate updates.

Thanks , i moved to 1.9.0 and it worked !
Repository: pykeen/pykeen
Issue: Saving Checkpoints to S3 bucket
GitHub labels: enhancement

🛑 Checkpoints

### Problem Statement

While training on AWS sagemaker, it is expensive to keep the checkpoints on the notebook. Being able to upload checkpoints directly to s3 bucket will save time.

### Describe the solution you'd like

in check points, allow s3 bucket urls 

```python
from pykeen.pipeline import pipeline

result = pipeline(
    model="transe",
    dataset="nations",
    training_kwargs=dict(
        num_epochs=10,
        checkpoint_name="test_checkpoint.pt",
        checkpoint_directory="s3://bucket/checkpoints/",
    ),
)
```



### Describe alternatives you've considered

Uploading checkpoints at the end of training for backup.

### Additional information

_No response_

### Issue Template Checks

- [X] This is not a bug report (use a different issue template if it is)
- [X] This is not a question (use the [discussions](https://github.com/pykeen/pykeen/discussions/new) forum instead)

Hi @mhmgad ,

we currently have [pykeen.pipeline.PipelineResult.save_to_s3](https://pykeen.readthedocs.io/en/stable/api/pykeen.pipeline.PipelineResult.html#pykeen.pipeline.PipelineResult.save_to_s3) to upload the final results to `s3`, but no support for checkpoints yet.

We would be happy to see a PR for this. The place to start from would be this method [`pykeen.training.training_loop.py:TrainingLoop._save_state`](https://github.com/pykeen/pykeen/blob/026cf4cb6f9917997b3b251ce4a12259560be04d/src/pykeen/training/training_loop.py#L1080-L1086).
Repository: pykeen/pykeen
Issue: Load pipeline result from save_to_directory
Root cause: Unimplemented Feature Gap

I trained a TransE model with my own data like so:

```
tf = TriplesFactory.from_path("graph.txt")
training, testing = tf.split()
result = pipeline(
    training=training,
    testing=testing,
    model='TransE',
    epochs=30, 
)
result.save_to_directory('doctests/test_unstratified_transe')
```

Now I need to load the pipeline result including the model, entity names, training triples, etc. so that I can pick up where I left off.

I imagined there would be a matching method, something like `load_from_directory` that loads the model and the entities automatically. I can't see anything like this and the documentation or older issues is not really helping either.

I get that I could load use `torch.load('trained_model.pkl')` but it wouldn't give me the correct class (`TransE`) and I still don't know how to load the entity names form `entity_to_id.tsv.gz`.

Does anyone have a piece of code that completely restores the results from the directory, as if it was in the memory when the training completed?
Repository: pykeen/pykeen
Issue: Reload Pipeline Results From Directory
Root cause: Unimplemented Feature Gap

### Problem Statement

Reloading the [`pykeen.pipeline.PipelineResult`](https://pykeen.readthedocs.io/en/latest/api/pykeen.pipeline.PipelineResult.html#pykeen.pipeline.PipelineResult) object from a directory seems to be frequently desired, cf., e.g.,

- https://github.com/pykeen/pykeen/discussions/1181#discussioncomment-4382887
- https://github.com/pykeen/pykeen/discussions/1189




### Describe the solution you'd like

Add a classmethod, e.g., `load_from_directory` to [`pykeen.pipeline.PipelineResult`](https://pykeen.readthedocs.io/en/latest/api/pykeen.pipeline.PipelineResult.html#pipelineresult) which allows to restore the object from the information stored in the directory.

### Describe alternatives you've considered

An alternative could be to improve the documentation on how to load the relevant parts into, e.g., pandas Dataframes.

### Additional information

- The serializaton format is described here: https://pykeen.readthedocs.io/en/latest/api/pykeen.pipeline.PipelineResult.html#pykeen.pipeline.PipelineResult.save_to_directory

### Issue Template Checks

- [X] This is not a bug report (use a different issue template if it is)
- [X] This is not a question (use the [discussions](https://github.com/pykeen/pykeen/discussions/new) forum instead)

Hello! I was looking into this, and I already have an idea about how to recover the `PipelineResult` from the files saved to the directory.

How should we treat the following attributes, whose values are not saved to the results directory?
* `random_seed`
*  `training_loop`
*  `configuration`
*  `version`
*  `git_hash`

Everything else can be recovered (I've just a doubt about how to recover the stopper, while I've already seen how to do that for the other attributes). Thank you for your help!

Can we make them optional (i.e., allow providing `None` for those in case they are not present)? This would allow to load results which have been saved before the current change.

For future `save_to_directory` calls, we could store them, too.
Repository: pykeen/pykeen
Issue: What's the meaning of the second return value(mask) of`NegativeSampler.sample()`

Hi everyone,

I have an issue when I take the following codes:

```python
'''
positive_triplets are generated in advance and look like:
tensor([[  0,   6,   6],
        [  0,   7,   7],
        [  0,   8,   8],
        [  0,   9,   9]])
'''

positive_triplets = ...
negative_sampler = BasicNegativeSampler(
    mapped_triples=positive_triplets, 
    num_negs_per_pos=2, filtered=True, 
    filterer=BloomFilterer, 
    filterer_kwargs={'error_rate': 1},
)
negative_triplets, mask = negative_sampler.sample(positive_triplets)
```
When I run codes above, I get `mask` all `False`. However, I found that triplets in `negative_triplets` are still true negative. For example, `negative_triplets[0, :, :]` exactly corrupt one head or one tail of `positive_triplets[0, :]`. But the masks of `negative_triplets[0, :, :]` are all `False`. Thus I'm wondering what `mask` means?

Also I'm wondering what the difference is between `sample()` and `corrupt_batch()`? From the doc I suppose the former use filterer while the latter does not use it. Do I understand it correctly?

Thanks.

Hi @LspongebobJH ,

did you already check https://pykeen.readthedocs.io/en/stable/reference/negative_sampling.html#module-pykeen.sampling.filtering ?

In your example, you chose the default filterer, which is fast, but only approximate. The trade-off between speed and accuracy can be tuned via the `error_rate` parameter. In your example, you chose an allowed error rate of 100%, which essentially disables the filterer.

Hi @mberr ,

Thanks for your reply. I understand your point. But my question is, when I set the `error_rate` to 100%, I suppose that all sampled negative triplets should be **false negative**, am I right? But I still get quite a lot of **true negative** triplets. And also, does the `False` in the returned mask means the corresponding negative triplet is a **false negative** ? BTW, I also raise another question about `sample()` and `corrupt_batch()` above, and just wondering if I understand that correctly. 

Thanks

> But my question is, when I set the error_rate to 100%, I suppose that all sampled negative triplets should be false negative, am I right?

No, the error rate is the maximum permitted error rate, cf. https://en.wikipedia.org/wiki/Bloom_filter for the underlying technique. To quote from there

> [False positive](https://en.wikipedia.org/wiki/Type_I_and_type_II_errors) matches are possible, but [false negatives](https://en.wikipedia.org/wiki/Type_I_and_type_II_errors) are not – in other words, a query returns either "possibly in set" or "definitely not in set