Repository: pykeen/pykeen
Issue: How to run replicate_best_pipeline with custom dataset?
Root cause: Documentation Deficiency

### What is your question

First of all: Thanks a lot for this awesome library :hugs: 

I'm confused by the API of `HpoPipelineResult.replicate_best_pipeline`: How am I supposed to pass in the training data it wants?

The following minimal example illustrates my confusion:

```python
import numpy as np

from pykeen.triples import TriplesFactory
from pykeen.datasets import Dataset
from pykeen.hpo import hpo_pipeline

# in my actual use case I would create the `Dataset` object from triples from neo4j
dummy_dataset = Dataset.from_tf(
    TriplesFactory.from_labeled_triples(
        np.random.randint(low=0, high=10, size=(100, 3)).astype(str)
    )
)

pipeline_result = hpo_pipeline(
    dataset=dummy_dataset,
    model="TransE",
    n_trials=2,
)

pipeline_result.replicate_best_pipeline(
    directory="dummy_transe_best_model",
    replicates=1,
)
```
Hyper-parameter optimization finishes successfully ("...Trial 1 finished with value: 0.06748466257668717 and parameters: {'model.embedding_dim': 64, 'model.scoring_fct_norm': 2, 'loss.margin': 2.3255194116052733, 'optimizer.lr': 0.07626777419512755, 'negative_sampler.num_negs_per_pos': 1, 'training.num_epochs': 200, 'training.batch_size': 732}. Best is trial 1 with value: 0.06748466257668717.
WARNING:pykeen.pipeline.api:No random seed is specified. Setting to 475292796.")

But the `replicate_best_pipeline` fails.

**Expected behavior**
 `replicate_best_pipeline` reruns model training with best parameters found during HPO and saves resulting model to "dummy_transh_best_model".

**Actual behavior**
Raises `ValueError: Must specify either dataset or both training/testing triples factories`

### Environment

```
| Key             | Value                    |
|-----------------|--------------------------|
| OS              | posix                    |
| Platform        | Linux                    |
| Release         | 5.13.0-28-generic        |
| Time            | Thu Feb 10 12:06:05 2022 |
| Python          | 3.8.12                   |
| PyKEEN          | 1.7.0                    |
| PyKEEN Hash     | UNHASHED                 |
| PyKEEN Branch   |                          |
| PyTorch         | 1.10.2                   |
| CUDA Available? | true                     |
| CUDA Version    | 11.3                     |
| cuDNN Version   | 8200                     |

```

### Issue Template Checks

- [X] This is not a bug report (use a different issue template if it is)
- [X] This is not a feature request (use a different issue template if it is)
- [X] I've read the text explaining why including environment information is important and understand if I omit this information that my issue will be dismissed

The full stacktrace is here
```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/tmp/ipykernel_2959262/2427177293.py in <module>
     19 )
     20 
---> 21 pipeline_result.replicate_best_pipeline(
     22     directory="dummy_transh_best_model",
     23     replicates=1,

~/anaconda3/envs/pykeen/lib/python3.8/site-packages/pykeen/hpo/hpo.py in replicate_best_pipeline(self, directory, replicates, move_to_cpu, save_replicates)
    456             raise ValueError("use_testing_data not be set in the configuration at at all!")
    457 
--> 458         replicate_pipeline_from_config(
    459             config=config,
    460             directory=directory,

~/anaconda3/envs/pykeen/lib/python3.8/site-packages/pykeen/pipeline/api.py in replicate_pipeline_from_config(config, directory, replicates, move_to_cpu, save_replicates, **kwargs)
    514     """
    515     pipeline_results = (pipeline_from_config(config, **kwargs) for _ in range(replicates))
--> 516     save_pipeline_results_to_directory(
    517         config=config,
    518         directory=directory,

~/anaconda3/envs/pykeen/lib/python3.8/site-packages/pykeen/pipeline/api.py in save_pipeline_results_to_directory(config, directory, pipeline_results, move_to_cpu, save_metadata, save_replicates, width)
    646     # TODO: we could have multiple results, if we get access to the raw results (e.g. in the pykeen benchmarking paper)
    647     result_comparator.add_original_result(result=config.get("results", {}))
--> 648     for i, pipeline_result in enumerate(pipeline_results):
    649         replicate_directory = replicates_directory.joinpath(f"replicate-{i:0{width}}")
    650         replicate_directory.mkdir(exist_ok=True, parents=True)

~/anaconda3/envs/pykeen/lib/python3.8/site-packages/pykeen/pipeline/api.py in <genexpr>(.0)
    513     :param kwargs: Keyword arguments to be passed through to :func:`pipeline_from_config`.
    514     """
--> 515     pipeline_results = (pipeline_from_config(config, **kwargs) for _ in range(replicates))
    516     save_pipeline_results_to_directory(
    517         config=config,

~/anaconda3/envs/pykeen/lib/python3.8/site-packages/pykeen/pipeline/api.py in pipeline_from_config(config, **kwargs)
    707         logger.info(f"Running: {title}")
    708 
--> 709     return pipeline(
    710         metadata=metadata,
    711         **pipeline_kwargs,

~/anaconda3/envs/pykeen/lib/python3.8/site-packages/pykeen/pipeline/api.py in pipeline(dataset, dataset_kwargs, training, testing, validation, evaluation_entity_whitelist, evaluation_relation_whitelist, model, model_kwargs, interaction, interaction_kwargs, dimensions, loss, loss_kwargs, regularizer, regularizer_kwargs, optimizer, optimizer_kwargs, clear_optimizer, lr_scheduler, lr_scheduler_kwargs, training_loop, training_loop_kwargs, negative_sampler, negative_sampler_kwargs, epochs, training_kwargs, stopper, stopper_kwargs, evaluator, evaluator_kwargs, evaluation_kwargs, result_tracker, result_tracker_kwargs, metadata, device, random_seed, use_testing_data, evaluation_fallback, filter_validation_when_testing, use_tqdm)
   1001     _device: torch.device = resolve_device(device)
   1002 
-> 1003     dataset_instance: Dataset = get_dataset(
   1004         dataset=dataset,
   1005         dataset_kwargs=dataset_kwargs,

~/anaconda3/envs/pykeen/lib/python3.8/site-packages/pykeen/datasets/__init__.py in get_dataset(dataset, dataset_kwargs, training, testing, validation)
    134     """
    135     if dataset is None and (training is None or testing is None):
--> 136         raise ValueError("Must specify either dataset or both training/testing triples factories")
    137 
    138     if dataset is not None and (training is not None or testing is not None):

ValueError: Must specify either dataset or both training/testing triples factories
```

The HPO pipeline was originally written to support our large-scale benchmarking study, so it's mostly tailored to built-in datasets. We've added some support for custom datasets, but tbh it's quite complicated since we had to do a lot of tricks to get the original architecture not to leak memory when running on a potentially shared GPU.

Note that the replication pipeline tries to reload the configuration for the best model, which doesn't adequately describe how to load and run the pipeline on custom datasets. Unfortunately, I don't have a lot of time to support this kind of custom application. A few things you can do in the mean time:

1. Read the tutorial on adding a custom dataset via an entrypoint
2. Try and hack in a way to give extra custom configuration in `replicate_best_pipeline` (or see if we already did that). A PR with a fix would be welcome!

Awesome, thanks a lot for the quick response! I've just been starting to use pykeen so I'll first look around a bit and see how far I get with your suggestion 1.

I'll close for now and reopen in case I find an easy fix or, maybe, an improvement to the docs which would help someone in my position

> 1. Read the tutorial on adding a custom dataset via an entrypoint

Linking the relevant part of the documentation here for reference: https://pykeen.readthedocs.io/en/stable/extending/datasets.html