Repository: pykeen/pykeen
Issue: Classification evaluator gets empty list during batch size search
Root cause: Validation Gap

### Describe the bug

I am training a model using "classification" evaluator using the following code

```python
from pykeen.pipeline import pipeline
result = pipeline(
    model='PairRE',
    dataset='Nations',
    stopper='early',
    epochs=100,
    dimensions=256,
    evaluator="classification"
)
```

The evaluator first logs:

```
> INFO:pykeen.evaluation.evaluator:Starting batch_size search for evaluation now...
> /usr/local/lib/python3.8/dist-packages/rexmex/metrics/classification.py:484: RuntimeWarning: invalid value encountered in power
>   pthr = (fpr**0.5) / (fpr**0.5 + tpr**0.5)
> /usr/local/lib/python3.8/dist-packages/rexmex/metrics/classification.py:547: RuntimeWarning: invalid value encountered in power
>   fm = (ppv * tpr) ** 0.5
> INFO:pykeen.evaluation.evaluator:Concluded batch_size search with batch_size=256.
```

And then breaks with the following output:

```
> ValueError                                Traceback (most recent call last)
> [<ipython-input-6-3d845fc18180>](https://localhost:8080/#) in <module>
>       1 from pykeen.pipeline import pipeline
>       2 
> ----> 3 result = pipeline(
>       4     training=training,
>       5     testing=testing,
> 
> 7 frames
> [/usr/local/lib/python3.8/dist-packages/pykeen/evaluation/classification_evaluator.py](https://localhost:8080/#) in finalize(self)
>      98         # we need to retrieve scores and masks using the exact same key order.
>      99         all_keys = list(self.all_scores.keys())
> --> 100         y_score = np.concatenate([self.all_scores[k] for k in all_keys], axis=0).flatten()
>     101         y_true = np.concatenate([self.all_positives[k] for k in all_keys], axis=0).flatten()
>     102 
> 
> <__array_function__ internals> in concatenate(*args, **kwargs)
> 
> ValueError: need at least one array to concatenate
```

I have then tried to reproduce on a Nations dataset, and I get the same runtime error but a different end error:

```
> KeyError                                  Traceback (most recent call last)
> [<ipython-input-3-ff3f4890c1c1>](https://localhost:8080/#) in <module>
>       1 from pykeen.pipeline import pipeline
> ----> 2 result = pipeline(
>       3     model='PairRE',
>       4     dataset='Nations',
>       5     stopper='early',
> 
> 6 frames
> [/usr/local/lib/python3.8/dist-packages/pykeen/evaluation/classification_evaluator.py](https://localhost:8080/#) in get_metric(self, name)
>      41     # docstr-coverage: inherited
>      42     def get_metric(self, name: str) -> float:  # noqa: D102
> ---> 43         return self.data[name]
>      44 
>      45 
> 
> KeyError: 'hits_at_k'
```


### Environment

| Key             | Value                    |
|-----------------|--------------------------|
| OS              | posix                    |
| Platform        | Linux                    |
| Release         | 5.10.147+                |
| Time            | Tue Jan 10 10:29:43 2023 |
| Python          | 3.8.16                   |
| PyKEEN          | 1.9.0                    |
| PyKEEN Hash     | UNHASHED                 |
| PyKEEN Branch   |                          |
| PyTorch         | 1.13.0+cu116             |
| CUDA Available? | true                     |
| CUDA Version    | 11.6                     |
| cuDNN Version   | 8302                     |

### Additional information

_No response_

### Issue Template Checks

- [X] This is not a feature request (use a different issue template if it is)
- [X] This is not a question (use the [discussions](https://github.com/pykeen/pykeen/discussions/new) forum instead)
- [X] I've read the text explaining why including environment information is important and understand if I omit this information that my issue will be dismissed

The first error sounds a lot like https://github.com/pykeen/pykeen/issues/819.

For the second, you need to adjust the default metrics used for early stopping to one avaiable in the classification evaluator, cf. https://github.com/pykeen/pykeen#metrics

The first error only appears when I use `evaluator="classification"`, with the default evaluation it works fine.
I would like to use this example in my book, and using only CPU seems to work, however it takes 1.5h for the first 10 epoch, while GPU takes like 2 minutes.
Anyway around this?

p.s. Actually it is the evaluation part that is taking forever. I use auc_roc. Are there other classification metrics that would be faster?

If you want to do early stopping on roc_auc, you need to tell this to the stopper via `stopper_kwargs`, e.g.,

```python
from pykeen.pipeline import pipeline
result = pipeline(
    model='PairRE',
    dataset='Nations',
    stopper='early',
    epochs=100,
    dimensions=256,
    evaluator="classification"
    stopper_kwargs=dict(metric="roc_auc_score"),  # <-- this line is new
)
```

Other possible metrics with the classification evaluator are
* "accuracy_score"
* "average_precision_score"
* "balanced_accuracy_score"
* "diagnostic_odds_ratio"
* "f1_score"
* "false_discovery_rate"
* "false_negative_rate"
* "false_omission_rate"
* "false_positive_rate"
* "fowlkes_mallows_index"
* "informedness"
* "markedness"
* "matthews_correlation_coefficient"
* "negative_likelihood_ratio"
* "negative_predictive_value"
* "positive_likelihood_ratio"
* "positive_predictive_value"
* "prevalence_threshold"
* "roc_auc_score"
* "threat_score"
* "true_negative_rate"
* "true_positive_rate"

Using correct classification metric for early stopper still doesn't work on GPU.
Is it possible to somehow disable automatic memory optimization for GPU as that is the problem if I understand https://github.com/pykeen/pykeen/issues/819 correctly.

I don't want to trouble readers with PyKEEN internals and would like the example to work on both GPU and CPU, so I'll do rank-based early stopper which works on both CPU and GPU. The idea was that people like typical metrics like accuracy, f1-score, AUC-ROC to evaluate their models. The book notebook for is available if it can help to reproduce GPU problems with roc_auc_score stopper: https://github.com/tomasonjo/graphs-network-science/blob/main/notebooks/Chapter%2011.ipynb

> Using correct classification metric for early stopper still doesn't work on GPU. Is it possible to somehow disable automatic memory optimization for GPU as that is the problem if I understand #819 correctly.
> 
> I don't want to trouble readers with PyKEEN internals and would like the example to work on both GPU and CPU, so I'll do rank-based early stopper which works on both CPU and GPU. The idea was that people like typical metrics like accuracy, f1-score, AUC-ROC to evaluate their models. The book notebook for is available if it can help to reproduce GPU problems with roc_auc_score stopper: https://github.com/tomasonjo/graphs-network-science/blob/main/notebooks/Chapter%2011.ipynb

Yes, you can manually set the batch size, too.

```python
from pykeen.pipeline import pipeline
result = pipeline(
    model='PairRE',
    dataset='Nations',
    stopper='early',
    epochs=100,
    dimensions=256,
    evaluator="classification",
    evaluation_kwargs=dict(batch_size=32),  # <-- this line is new
    # notice that this is evaluaTION_kwargs, and not evaluaTOR_kwargs, which does exist, too
    stopper_kwargs=dict(metric="roc_auc_score"),
)
```

No luck, the error persists on GPU. I am out of ideas what could be wrong.

I currently do not have a GPU test machine at hand, but could you check whether https://github.com/pykeen/pykeen/pull/1195 fixes this?

```
INFO:pykeen.evaluation.evaluator:Starting batch_size search for evaluation now...
/usr/local/lib/python3.8/dist-packages/rexmex/metrics/classification.py:484: RuntimeWarning: invalid value encountered in power
  pthr = (fpr**0.5) / (fpr**0.5 + tpr**0.5)
/usr/local/lib/python3.8/dist-packages/rexmex/metrics/classification.py:547: RuntimeWarning: invalid value encountered in power
  fm = (ppv * tpr) ** 0.5
INFO:pykeen.evaluation.evaluator:Concluded batch_size search with batch_size=256.
/usr/local/lib/python3.8/dist-packages/rexmex/metrics/classification.py:127: RuntimeWarning: invalid value encountered in float_scalars
  tnr = tn / n
/usr/local/lib/python3.8/dist-packages/rexmex/metrics/classification.py:195: RuntimeWarning: invalid value encountered in float_scalars
  tpr = tp / p
/usr/local/lib/python3.8/dist-packages/rexmex/metrics/classification.py:245: RuntimeWarning: invalid value encountered in float_scalars
  ppv = tp / (tp + fp)
/usr/local/lib/python3.8/dist-packages/rexmex/metrics/classification.py:286: RuntimeWarning: invalid value encountered in float_scalars
  npv = tn / (tn + fn)
/usr/local/lib/python3.8/dist-packages/rexmex/metrics/classification.py:324: RuntimeWarning: invalid value encountered in float_scalars
  fnr = fn / p
/usr/local/lib/python3.8/dist-packages/rexmex/metrics/classification.py:362: RuntimeWarning: invalid value encountered in float_scalars
  fpr = fp / n
/usr/local/lib/python3.8/dist-packages/rexmex/metrics/classification.py:385: RuntimeWarning: invalid value encountered in float_scalars
  fdr = fp / (fp + tp)
/usr/local/lib/python3.8/dist-packages/rexmex/metrics/classification.py:408: RuntimeWarning: invalid value encountered in float_scalars
  fomr = fn / (fn + tn)
/usr/local/lib/python3.8/dist-packages/rexmex/metrics/classification.py:508: RuntimeWarning: invalid value encountered in float_scalars
  ts = tp / (tp + fn + fp)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-6-ae80c553852e>](https://localhost:8080/#) in <module>
      1 from pykeen.pipeline import pipeline
      2 
----> 3 result = pipeline(
      4     training=training,
      5     testing=testing,

13 frames
[/usr/local/lib/python3.8/dist-packages/sklearn/utils/validation.py](https://localhost:8080/#) in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator)
    803         n_samples = _num_samples(array)
    804         if n_samples < ensure_min_samples:
--> 805             raise ValueError(
    806                 "Found array with %d sample(s) (shape=%s) while a"
    807                 " minimum of %d is required%s."

ValueError: Found array with 0 sample(s) (shape=(0,)) while a minimum of 1 is required.
```

btw.. Google Colab offers free GPU, I use that.
Could it be due to a negative base, i.e. https://cumsum.wordpress.com/2020/05/03/runtimewarning-invalid-value-encountered-in-power-numpy/

https://github.com/pykeen/pykeen/pull/1195/commits/238328003d3be3720ba2c6eb07daedae2a0c1a6c should have fixed the issue, though I am a bit worried that it might shadow some underlying problem 🤔 

I fixed the underlying problem in #1195 🙂 This will get to `master` branch soon.

Awesome, great! I tested the previous commit and it passed through with the same runtime errors, but it worked.
Just one last thing. The classification evaluation takes 1 hour, where rank-based evaluation takes like 3 minutes. Is this something normal?

Btw... do you guys take any beer donations?

> Awesome, great! I tested the previous commit and it passed through with the same runtime errors, but it worked. Just one last thing. The classification evaluation takes 1 hour, where rank-based evaluation takes like 3 minutes. Is this something normal?
> 
> Btw... do you guys take any beer donations?

We were just looking into setting up GitHub sponsors as a way of accepting such beer donations. It looks like you already found it! That is very gracious of you, thank you :)