Repository: pykeen/pykeen
Issue: Unexpected Early Stopping
Root cause: Documentation Deficiency

### Describe the bug

The algorithm stopped even though the "current" evaluation is better than the previous ones. One would expect that the training will keep going on.

### How to reproduce

```python
from pykeen.datasets import Kinships
from pykeen.evaluation import RankBasedEvaluator
from pykeen.models import DistMult
from pykeen.pipeline import pipeline

data = Kinships()
res = pipeline(
    model=DistMult,
    model_kwargs={"random_seed": 0},
    training=data.training,
    testing=data.testing,
    validation=data.validation,
    dimensions=50,
    epochs=100,
    random_seed=0,
    stopper="early",
    stopper_kwargs={"frequency": 5, "patience": 2},
    evaluator=RankBasedEvaluator(),
)
```

### Environment

| Key             | Value                    |
|-----------------|--------------------------|
| OS              | posix                    |
| Platform        | Linux                    |
| Release         | 5.10.147+                |
| Time            | Mon Feb 20 18:03:37 2023 |
| Python          | 3.8.10                   |
| PyKEEN          | 1.10.1-dev               |
| PyKEEN Hash     | UNHASHED                 |
| PyKEEN Branch   |                          |
| PyTorch         | 1.13.1+cu116             |
| CUDA Available? | false                    |
| CUDA Version    | 11.6                     |
| cuDNN Version   | 8302                     |

This is on Google Colab.

### Additional information

Training output:
```
WARNING:pykeen.utils:No cuda devices were available. The model runs on CPU
INFO:pykeen.pipeline.api:Using device: None
INFO:pykeen.stoppers.early_stopping:Inferred checkpoint path for best model weights: /root/.data/pykeen/checkpoints/best-model-weights-e34d79cb-fc7a-471c-883d-43e29078f34d.pt
Training epochs on cpu: 34%
34/100 [00:17<00:30, 2.20epoch/s, loss=0.43, prev_loss=0.442]
INFO:pykeen.evaluation.evaluator:Currently automatic memory optimization only supports GPUs, but you're using a CPU. Therefore, the batch_size will be set to the default value.
INFO:pykeen.evaluation.evaluator:No evaluation batch_size provided. Setting batch_size to '32'.
INFO:pykeen.evaluation.evaluator:Evaluation took 0.22s seconds
INFO:pykeen.stoppers.early_stopping:New best result at epoch 5: 0.12125468164794008. Saved model weights to /root/.data/pykeen/checkpoints/best-model-weights-e34d79cb-fc7a-471c-883d-43e29078f34d.pt
INFO:pykeen.training.training_loop:=> Saved checkpoint after having finished epoch 5.
INFO:pykeen.evaluation.evaluator:Currently automatic memory optimization only supports GPUs, but you're using a CPU. Therefore, the batch_size will be set to the default value.
INFO:pykeen.evaluation.evaluator:No evaluation batch_size provided. Setting batch_size to '32'.
INFO:pykeen.evaluation.evaluator:Evaluation took 0.21s seconds
INFO:pykeen.stoppers.early_stopping:New best result at epoch 10: 0.24204119850187267. Saved model weights to /root/.data/pykeen/checkpoints/best-model-weights-e34d79cb-fc7a-471c-883d-43e29078f34d.pt
INFO:pykeen.training.training_loop:=> Saved checkpoint after having finished epoch 10.
INFO:pykeen.evaluation.evaluator:Currently automatic memory optimization only supports GPUs, but you're using a CPU. Therefore, the batch_size will be set to the default value.
INFO:pykeen.evaluation.evaluator:No evaluation batch_size provided. Setting batch_size to '32'.
INFO:pykeen.evaluation.evaluator:Evaluation took 0.21s seconds
INFO:pykeen.stoppers.early_stopping:New best result at epoch 15: 0.300561797752809. Saved model weights to /root/.data/pykeen/checkpoints/best-model-weights-e34d79cb-fc7a-471c-883d-43e29078f34d.pt
INFO:pykeen.training.training_loop:=> Saved checkpoint after having finished epoch 15.
INFO:pykeen.evaluation.evaluator:Currently automatic memory optimization only supports GPUs, but you're using a CPU. Therefore, the batch_size will be set to the default value.
INFO:pykeen.evaluation.evaluator:No evaluation batch_size provided. Setting batch_size to '32'.
INFO:pykeen.evaluation.evaluator:Evaluation took 0.23s seconds
INFO:pykeen.stoppers.early_stopping:New best result at epoch 20: 0.6591760299625468. Saved model weights to /root/.data/pykeen/checkpoints/best-model-weights-e34d79cb-fc7a-471c-883d-43e29078f34d.pt
INFO:pykeen.training.training_loop:=> Saved checkpoint after having finished epoch 20.
INFO:pykeen.evaluation.evaluator:Currently automatic memory optimization only supports GPUs, but you're using a CPU. Therefore, the batch_size will be set to the default value.
INFO:pykeen.evaluation.evaluator:No evaluation batch_size provided. Setting batch_size to '32'.
INFO:pykeen.evaluation.evaluator:Evaluation took 0.35s seconds
INFO:pykeen.stoppers.early_stopping:New best result at epoch 25: 0.6999063670411985. Saved model weights to /root/.data/pykeen/checkpoints/best-model-weights-e34d79cb-fc7a-471c-883d-43e29078f34d.pt
INFO:pykeen.training.training_loop:=> Saved checkpoint after having finished epoch 25.
INFO:pykeen.evaluation.evaluator:Currently automatic memory optimization only supports GPUs, but you're using a CPU. Therefore, the batch_size will be set to the default value.
INFO:pykeen.evaluation.evaluator:No evaluation batch_size provided. Setting batch_size to '32'.
INFO:pykeen.evaluation.evaluator:Evaluation took 0.20s seconds
INFO:pykeen.evaluation.evaluator:Currently automatic memory optimization only supports GPUs, but you're using a CPU. Therefore, the batch_size will be set to the default value.
INFO:pykeen.evaluation.evaluator:No evaluation batch_size provided. Setting batch_size to '32'.
INFO:pykeen.evaluation.evaluator:Evaluation took 0.22s seconds
INFO:pykeen.stoppers.early_stopping:Stopping early at epoch 35. The best result 0.6999063670411985 occurred at epoch 25.
INFO:pykeen.stoppers.early_stopping:Re-loading weights from best epoch from /root/.data/pykeen/checkpoints/best-model-weights-e34d79cb-fc7a-471c-883d-43e29078f34d.pt
INFO:pykeen.evaluation.evaluator:Currently automatic memory optimization only supports GPUs, but you're using a CPU. Therefore, the batch_size will be set to the default value.
INFO:pykeen.evaluation.evaluator:No evaluation batch_size provided. Setting batch_size to '32'.
Evaluating on cpu: 100%
1.07k/1.07k [00:00<00:00, 4.88ktriple/s]
INFO:pykeen.evaluation.evaluator:Evaluation took 0.31s seconds
CPU times: user 17.7 s, sys: 215 ms, total: 17.9 s
Wall time: 18.5 s
```

Output of `res.stopper.results`:
```
[0.12125468164794008,
 0.24204119850187267,
 0.300561797752809,
 0.6591760299625468,
 0.6999063670411985,
 0.6853932584269663,
 0.7050561797752809]
```


### Issue Template Checks

- [X] This is not a feature request (use a different issue template if it is)
- [X] This is not a question (use the [discussions](https://github.com/pykeen/pykeen/discussions/new) forum instead)
- [X] I've read the text explaining why including environment information is important and understand if I omit this information that my issue will be dismissed

Hi @wendywangwwt ,

by default, the early stopper uses a minimum relative difference of 1% to consider a decrease an improvement. You can change that behavior by explicitly providing a value for `relative_delta` in the `stopper_kwargs`, e.g., 
```python
stopper_kwargs={'frequency':5, 'patience': 2, 'relative_delta': 0.0}, 
```
A description of all parameters can be found at the documentation of the `EarlyStopper` class: https://pykeen.readthedocs.io/en/stable/reference/stoppers.html#pykeen.stoppers.EarlyStopper

@mberr Thank you for the advice! Sorry for missing this parameter and yes, I can confirm that after setting it to 0, the training keeps going as expected.

Intuitively though, I'd expect `relative_delta` to be defaulted to 0 🤔 

> Intuitively though, I'd expect relative_delta to be defaulted to 0 🤔

I think the reason behind this default choice was that we want to prevent not stopping just because the metric naturally fluctuates slightly around a minimum (and with a large enough patience, we occasionally observe slight improvements "just by chance").

It may be worth to update the early stopping logging messages though to at least indicate that the stop happened with a `relative_delta` rule.

update: sorry the early stopping is correct 😓 

-----
<del>Reopening the issue as this doesn't fix the unexpected early stopping for `ComplEx` - </del>

<del>#### Reproduction Code</del>

<del>#### Output</del>

Apart from this, it seems that for ComplEx, argument `dimensions` in `pipeline()` is not used. I have to control it by passing `embedding_dim` in `model_kwargs`.

P.S. the result is surprisingly bad.. I'm still trying to figure out why :/

> P.S. the result is surprisingly bad.. I'm still trying to figure out why :/

bad in what terms? From the metrics?