Repository: pykeen/pykeen
Issue: Skipping Validation step of Pipeline AND loading from Checkpoints
Root cause: Documentation Deficiency

### What is your question

Hello, strictly speaking these are two questions that have arisen from trying to deal with the same issue. I'd appreciate any input anyone can give.

I am trying to train a number of different models on a set of triples with parameters passed. The pipeline is successful for smaller dimensions of embeddings (100). Ideally I wanted to train with a larger embedding dimension (200) and run the pipeline. The training completes fine, however the pipeline then goes into a validation step that overloads the GPU memory. I have not been able to understand from the documentation how I can either skip this evaluation step or dramatically reduce it.  I had looked into trying to use the `additional_filter_triples` parameter to the `evaluate()` function through `evaluation_kwargs`, and setting this to `[self.training.mapped_triples]`, however this did not seem to reduce the size of the evaluation.

I have a pipeline set up as follows:

```python
tracker = PythonResultTracker()
self.results = pipeline(
    training=self.training, # the dataset is loaded to a triples factory and then split using its split() method
    testing=self.testing,
    validation=self.validation,
    model=embed_algorithm, 
    epochs=num_epochs,
    model_kwargs=dict(
        embedding_dim=embed_dimension,
        ),
    evaluation_kwargs=dict(
        additional_filter_triples=[self.training.mapped_triples]
        ),
    training_kwargs=dict(
        batch_size = train_batch_size,
        checkpoint_name = checkpoint_file,
        checkpoint_on_failure = True,
        checkpoint_directory = os.path.join(data_dir,"embed_models","checkpoints"),
        ),
    result_tracker=tracker,
    filter_validation_when_testing=True,
    random_seed=111,
)
```

As a work around to the memory errors in the evaluation, I added a checkpoint to my models and loaded them from the checkpoint to perform the tail prediction on a dataset of head entities. I followed the advise on loading from a checkpoint, and adapted it to my case with the following method:

```python
def load_embeddings_from_checkpoint(self, filepath, model_type):
        checkpoint = torch.load(filepath)
        self.load_database() # this loads the unsplit dataset used for training to self.triples
        self.triples.with_labels(entity_to_id=checkpoint['entity_to_id_dict'], relation_to_id=checkpoint['relation_to_id_dict'],)
        model_class = globals()[model_type]
        self.model = model_class(triples_factory=self.triples)
        self.model.load_state_dict(checkpoint['model_state_dict'])
```

This appears to work fine, and I am able to load a model to use for tail prediction. However, when I compute the tail dataset using `predicted_tails = self.model.get_tail_prediction_df(head_label, rel_label, triples_factory = self.triples, remove_known=True)`,  i receive a CPU allocation error, stating out of memory. 

I am unsure why this method is needing to allocate so much memory on my CPU, as I understood the model was on the GPU, and I would have thought the CPU would only be needing to allocate for the size of the dataframe (the graph has ~200k nodes so the dataframe is not particularly large). I cannot seem to find a way at present in the documentation to  ensure the model is on the GPU.

So my main questions are:
- Is there a way to bypass the final evaluation step in a pipeline?
- How am I able to scale down the final evaluation step so it isnt computing across the whole graph?
- Am I loading the model from the checkpoint correctly?
- How can I guarantee the model is loaded on GPU?
- Are there any obvious causes of CPU Allocation error in the `get_tail_prediction_df()` step?

Thank you!

### Environment

| Key             | Value                    |
|-----------------|--------------------------|
| OS              | nt                       |
| Platform        | Windows                  |
| Release         | 10                       |
| Time            | Wed Jun 29 14:42:50 2022 |
| Python          | 3.9.7                    |
| PyKEEN          | 1.8.1                    |
| PyKEEN Hash     | UNHASHED                 |
| PyKEEN Branch   |                          |
| PyTorch         | 1.10.1                   |
| CUDA Available? | true                     |
| CUDA Version    | 11.3                     |
| cuDNN Version   | 8200                     |

### Issue Template Checks

- [X] This is not a bug report (use a different issue template if it is)
- [X] This is not a feature request (use a different issue template if it is)
- [X] I've read the text explaining why including environment information is important and understand if I omit this information that my issue will be dismissed

As additional info:

It seems that the out of memory error on CPU when running the tail predictions was only occurring when I was trying to use the ConvKB model. 

The out of memory on evaluation is happening for all models I have tried (currently using RotatE, ProjE, AutoSF and TuckER), but I am able to successfully load the checkpoints and make tail predictions. I noticed I did not specify the `embed_dim` in my code for loading from checkpoint, and this has been corrected.