Repository: pykeen/pykeen
Issue: ValueError: need at least one array to concatenate in rank_based_evaluator.py
Root cause: Validation Gap

### Describe the bug

When running a pipeline to train a model it calls to the `rank_based_evaluator.py script`. There, the line `c_ranks = np.concatenate([ranks_flat[side, rank_type] for side in sides])` outputs the following error:

```python-traceback
 File "<__array_function__ internals>", line 5, in concatenate
ValueError: need at least one array to concatenate
```


### How to reproduce

I haven't developed a script to make the error reproducible, but it happens when calling the pipeline function:

```python
results = pipeline(
    training=training,
    testing=testing,
    validation=validation,
    model=model,
    model_kwargs=dict(
        embedding_dim=embedding
    ),
    loss=loss,
    training_loop='sLCWA',
    negative_sampler='basic',
    result_tracker='tensorboard',
    result_tracker_kwargs=dict(
        experiment_path=logdir,
    ),
    training_kwargs=dict(
        num_epochs=epochs,
        batch_size=batch_size,
        sampler=sampler),
    stopper='early',
    stopper_kwargs=dict(
        frequency=5,
        patience=10,
        relative_delta=0.05,
        metric='adjusted_mean_rank_index',
    ),
    random_seed=42,
    device=device
)
```

### Environment

Unable to handle parameter in AutoSF: coefficients
| Key             | Value                       |
|-----------------|-----------------------------|
| OS              | posix                       |
| Platform        | Linux                       |
| Release         | 3.10.0-1160.36.2.el7.x86_64 |
| Time            | Mon Mar  7 17:27:26 2022    |
| Python          | 3.8.12                      |
| PyKEEN          | 1.7.1-dev                   |
| PyKEEN Hash     | UNHASHED                    |
| PyKEEN Branch   |                             |
| PyTorch         | 1.10.2                      |
| CUDA Available? | false                       |
| CUDA Version    | 10.2                        |
| cuDNN Version   | 7605                        |

### Additional information

I suspect that it's related with some recent update, because I have other environment in which the code runs perfectly. The numpy version is the same in both environments. My current environment is the following:

<details>

```
 Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main    conda-forge
_openmp_mutex             4.5                       1_gnu  
absl-py                   0.15.0             pyhd3eb1b0_0  
aiohttp                   3.8.1            py38h7f8727e_0  
aiosignal                 1.2.0              pyhd3eb1b0_0  
alembic                   1.7.6                    pypi_0    pypi
async-timeout             4.0.1              pyhd3eb1b0_0  
attrs                     21.4.0             pyhd3eb1b0_0  
autopage                  0.5.0                    pypi_0    pypi
blas                      1.0                         mkl    conda-forge
blinker                   1.4              py38h06a4308_0  
bottleneck                1.3.2            py38heb32a55_1  
brotlipy                  0.7.0           py38h497a2fe_1001    conda-forge
bzip2                     1.0.8                h7b6447c_0  
c-ares                    1.18.1               h7f8727e_0  
ca-certificates           2021.10.8            ha878542_0    conda-forge
cachetools                4.2.2              pyhd3eb1b0_0  
certifi                   2021.10.8        py38h578d9bd_1    conda-forge
cffi                      1.15.0           py38hd667e15_1  
charset-normalizer        2.0.12             pyhd8ed1ab_0    conda-forge
class-resolver            0.3.4                    pypi_0    pypi
click                     8.0.4            py38h06a4308_0  
click-default-group       1.2.2                    pypi_0    pypi
cliff                     3.10.1                   pypi_0    pypi
cmaes                     0.8.2                    pypi_0    pypi
cmd2                      2.4.0                    pypi_0    pypi
colorama                  0.4.4              pyh9f0ad1d_0    conda-forge
colorlog                  6.6.0                    pypi_0    pypi
cryptography              35.0.0           py38ha5dfef3_0    conda-forge
cudatoolkit               10.2.89              hfd86e86_1  
dataclasses               0.8                pyh6d0b6a4_7  
dataclasses-json          0.5.6                    pypi_0    pypi
decorator                 4.4.2                      py_0    conda-forge
docdata                   0.0.3                    pypi_0    pypi
docrep                    0.3.2                    pypi_0    pypi
ffmpeg                    4.3                  hf484d3e_0    pytorch
freetype                  2.11.0               h70c0345_0  
frozenlist                1.2.0            py38h7f8727e_0  
giflib                    5.2.1                h7b6447c_0  
gmp                       6.2.1                h2531618_2  
gnutls                    3.6.15               he1e5248_0  
google-auth               1.33.0             pyhd3eb1b0_0  
google-auth-oauthlib      0.4.1                      py_2  
googledrivedownloader     0.4                pyhd3deb0d_1    conda-forge
greenlet                  1.1.2                    pypi_0    pypi
grpcio                    1.42.0           py38hce63b2e_0  
html5lib                  1.1                pyh9f0ad1d_0    conda-forge
idna                      3.3                pyhd8ed1ab_0    conda-forge
importlib-metadata        4.11.2                   pypi_0    pypi
importlib-resources       5.4.0                    pypi_0    pypi
inflect                   5.4.0                    pypi_0    pypi
intel-openmp              2021.4.0          h06a4308_3561  
isodate                   0.6.1              pyhd8ed1ab_0    conda-forge
jinja2                    3.0.3              pyhd8ed1ab_0    conda-forge
joblib                    1.1.0              pyhd8ed1ab_0    conda-forge
jpeg                      9d                   h7f8727e_0  
lame                      3.100                h7b6447c_0  
lcms2                     2.12                 h3be6417_0  
ld_impl_linux-64          2.35.1               h7274673_9  
libblas                   3.9.0            12_linux64_mkl    conda-forge
libcblas                  3.9.0            12_linux64_mkl    conda-forge
libffi                    3.3                  he6710b0_2  
libgcc-ng                 9.3.0               h5101ec6_17  
libgfortran-ng            7.5.0               h14aa051_20    conda-forge
libgfortran4              7.5.0               h14aa051_20    conda-forge
libgomp                   9.3.0               h5101ec6_17  
libiconv                  1.15                 h63c8f33_5  
libidn2                   2.3.2                h7f8727e_0  
liblapack                 3.9.0            12_linux64_mkl    conda-forge
libpng                    1.6.37               hbc83047_0  
libprotobuf               3.19.1               h4ff587b_0  
libstdcxx-ng              9.3.0               hd4cf53a_17  
libtasn1                  4.16.0               h27cfd23_0  
libtiff                   4.2.0                h85742a9_0  
libunistring              0.9.10               h27cfd23_0  
libuv                     1.40.0               h7b6447c_0  
libwebp                   1.2.2                h55f646e_0  
libwebp-base              1.2.2                h7f8727e_0  
lz4-c                     1.9.3                h295c915_1  
mako                      1.1.6                    pypi_0    pypi
markdown                  3.3.4            py38h06a4308_0  
markupsafe                2.1.0                    pypi_0    pypi
marshmallow               3.14.1                   pypi_0    pypi
marshmallow-enum          1.5.1                    pypi_0    pypi
mkl                       2021.4.0           h06a4308_640  
mkl-service               2.4.0            py38h7f8727e_0  
mkl_fft                   1.3.1            py38hd3c417c_0  
mkl_random                1.2.2            py38h51133e4_0  
more-click                0.0.6                    pypi_0    pypi
more-itertools            8.12.0                   pypi_0    pypi
multidict                 5.2.0            py38h7f8727e_2  
mypy-extensions           0.4.3                    pypi_0    pypi
ncurses                   6.3                  h7f8727e_2  
nettle                    3.7.3                hbbd107a_1  
networkx                  2.5.1              pyhd8ed1ab_0    conda-forge
numexpr                   2.8.1            py38h6abb31d_0  
numpy                     1.20.3                   pypi_0    pypi
oauthlib                  3.1.0                      py_0  
openh264                  2.1.1                h4ff587b_0  
openssl                   1.1.1m               h7f8727e_0  
optuna                    2.10.0                   pypi_0    pypi
packaging                 21.3               pyhd3eb1b0_0  
pandas                    1.3.5                    pypi_0    pypi
pbr                       5.8.1                    pypi_0    pypi
pillow                    9.0.1            py38h22f2fdc_0  
pip                       21.2.4           py38h06a4308_0  
prettytable               3.2.0                    pypi_0    pypi
protobuf                  3.19.1           py38h295c915_0  
pyasn1                    0.4.8              pyhd3eb1b0_0  
pyasn1-modules            0.2.8                      py_0  
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pyg                       2.0.3           py38_torch_1.10.0_cu102    pyg
pyjwt                     1.7.1                    py38_0  
pykeen                    1.7.1.dev0               pypi_0    pypi
pyopenssl                 22.0.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.7              pyhd8ed1ab_0    conda-forge
pyperclip                 1.8.2                    pypi_0    pypi
pysocks                   1.7.1            py38h578d9bd_4    conda-forge
pystow                    0.4.0                    pypi_0    pypi
python                    3.8.12               h12debd9_0  
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-louvain            0.15               pyhd8ed1ab_1    conda-forge
python_abi                3.8                      2_cp38    conda-forge
pytorch                   1.10.2          py3.8_cuda10.2_cudnn7.6.5_0    pytorch
pytorch-cluster           1.5.9           py38_torch_1.10.0_cu102    pyg
pytorch-mutex             1.0                        cuda    pytorch
pytorch-scatter           2.0.9           py38_torch_1.10.0_cu102    pyg
pytorch-sparse            0.6.12          py38_torch_1.10.0_cu102    pyg
pytorch-spline-conv       1.2.1           py38_torch_1.10.0_cu102    pyg
pytz                      2021.3             pyhd8ed1ab_0    conda-forge
pyyaml                    6.0                      pypi_0    pypi
rdflib                    6.1.1              pyhd8ed1ab_0    conda-forge
readline                  8.1.2                h7f8727e_1  
requests                  2.27.1             pyhd8ed1ab_0    conda-forge
requests-oauthlib         1.3.0                      py_0  
rexmex                    0.1.0                    pypi_0    pypi
rsa                       4.7.2              pyhd3eb1b0_1  
scikit-learn              1.0.2            py38h51133e4_1  
scipy                     1.8.0                    pypi_0    pypi
setuptools                58.0.4           py38h06a4308_0  
six                       1.16.0             pyhd3eb1b0_1  
sklearn                   0.0                      pypi_0    pypi
sqlalchemy                1.4.32                   pypi_0    pypi
sqlite                    3.37.2               hc218d9a_0  
stevedore                 3.5.0                    pypi_0    pypi
tabulate                  0.8.9                    pypi_0    pypi
tensorboard               2.6.0                      py_1  
tensorboard-data-server   0.6.0            py38hca6d32c_0  
tensorboard-plugin-wit    1.6.0                      py_0  
threadpoolctl             3.1.0              pyh8a188c0_0    conda-forge
tk                        8.6.11               h1ccaba5_0  
torchaudio                0.10.2               py38_cu102    pytorch
torchvision               0.11.3               py38_cu102    pytorch
tqdm                      4.63.0             pyhd8ed1ab_0    conda-forge
typing-extensions         3.10.0.2             hd3eb1b0_0  
typing-inspect            0.7.1                    pypi_0    pypi
typing_extensions         3.10.0.2           pyh06a4308_0  
urllib3                   1.26.8             pyhd8ed1ab_1    conda-forge
wcwidth                   0.2.5                    pypi_0    pypi
webencodings              0.5.1                      py_1    conda-forge
werkzeug                  2.0.3              pyhd3eb1b0_0  
wheel                     0.37.1             pyhd3eb1b0_0  
wrapt                     1.13.3                   pypi_0    pypi
xz                        5.2.5                h7b6447c_0  
yacs                      0.1.8              pyhd8ed1ab_0    conda-forge
yaml                      0.2.5                h516909a_0    conda-forge
yarl                      1.6.3            py38h27cfd23_0  
zipp                      3.7.0              pyhd3eb1b0_0  
zlib                      1.2.11               h7f8727e_4  
zstd                      1.4.9                haebb681_0 
```
</details>

Hi @AlejandroTL ,

could you please post the full traceback so we can see the call stack?

Also, is one of the evaluation triples factories empty?

Hello @mberr, here it's the full traceback. I've just removed some paths for privacy.

```python-traceback
Traceback (most recent call last):
  File "knowldege_graph-/Train_KG.py", line 45, in <module>
    selection_train(training_dict)
  File "~/kge_train_utils.py", line 101, in selection_train
    train_routine(hyper_dict)
  File "~/kge_train_utils.py", line 128, in train_routine
    results = pipeline(
  File "/envs/my_env/lib/python3.8/site-packages/pykeen/pipeline/api.py", line 1237, in pipeline
    losses = training_loop_instance.train(
  File "/envs/my_env/lib/python3.8/site-packages/pykeen/training/training_loop.py", line 347, in train
    result = self._train(
  File "/envs/my_env/lib/python3.8/site-packages/pykeen/training/training_loop.py", line 733, in _train
    callback.post_epoch(epoch=epoch, epoch_loss=epoch_loss)
  File "/envs/my_env/lib/python3.8/site-packages/pykeen/training/callbacks.py", line 367, in post_epoch
    callback.post_epoch(epoch=epoch, epoch_loss=epoch_loss, **kwargs)
  File "/envs/my_env/lib/python3.8/site-packages/pykeen/training/callbacks.py", line 296, in post_epoch
    if self.stopper.should_stop(epoch):
  File "/envs/my_env/lib/python3.8/site-packages/pykeen/stoppers/early_stopping.py", line 205, in should_stop
    metric_results = self.evaluator.evaluate(
  File "/envs/my_env/lib/python3.8/site-packages/pykeen/evaluation/evaluator.py", line 187, in evaluate
    self.finalize()
  File "/envs/my_env/lib/python3.8/site-packages/pykeen/evaluation/rank_based_evaluator.py", line 263, in finalize
    result = RankBasedMetricResults.from_ranks(
  File "/envs/my_env/lib/python3.8/site-packages/pykeen/evaluation/rank_based_evaluator.py", line 96, in from_ranks
    for metric, (target, rank_type, ranks, num_candidates) in itertools.product(
  File "/envs/my_env/lib/python3.8/site-packages/pykeen/evaluation/rank_based_evaluator.py", line 69, in _iter_ranks
    c_ranks = np.concatenate([ranks_flat[side, rank_type] for side in sides])
  File "<__array_function__ internals>", line 5, in concatenate
ValueError: need at least one array to concatenate
```

And no, none of the evaluation triples factory is empty.

@AlejandroTL so the error comes from the early stopper's call to the evaluator. Could you verify that this script runs smooth in your setup? It should be mostly the same without a result tracker, and with a (small) builtin dataset instead of your local one.

```python
from pykeen.pipeline import pipeline

results = pipeline(
    dataset="nations",
    model="distmult",
    stopper="early",
    stopper_kwargs=dict(
        frequency=5,
        patience=10,
        relative_delta=0.05,
        metric="adjusted_mean_rank_index",
    ),
)
```

@mberr it appears exactly the same error. I do not copy the full trace because it's exactly the same as the one I copied yesterday.



You seem to have installed PyKEEN from source - could you update your installation with the current `master`, and see again whether https://github.com/pykeen/pykeen/issues/819#issuecomment-1060995463 works?

I have run `pip install git+https://github.com/pykeen/pykeen.git`. Is this the way to update the installation with the current `master`? I installed PyKeen in this way before too. If you mean something else, please indicate me how to do it (even if it's so silly, sorry).

> I have run `pip install git+https://github.com/pykeen/pykeen.git`. Is this the way to update the installation with the current `master`? I installed PyKeen in this way before too. If you mean something else, please indicate me how to do it (even if it's so silly, sorry).

I am not sure whether you need to provide the `-U` flag for updating the installation :thinking: 
```console
pip install -U git+https://github.com/pykeen/pykeen.git
```

You should see from the `pip` output whether it installs something or skips installation.

Already done it with the `-U` flag. Nothing installed, everything up to date. 
Same error :(

Okay, then let's try to explicitly uninstall PyKEEN first:
```console
pip uninstall pykeen
pip install -U git+https://github.com/pykeen/pykeen.git
```

Same error :(

For reference: this is the output of `pykeen.env()` where the script from https://github.com/pykeen/pykeen/issues/819#issuecomment-1060995463 runs smoothly.

| Key             | Value                    |
|-----------------|--------------------------|
| OS              | posix                    |
| Platform        | Linux                    |
| Release         | 5.13.0-30-generic        |
| Time            | Tue Mar  8 11:43:05 2022 |
| Python          | 3.9.7                    |
| PyKEEN          | 1.7.1-dev                |
| PyKEEN Hash     | 2fb430b2                 |
| PyKEEN Branch   | master                   |
| PyTorch         | 1.10.2+cpu               |
| CUDA Available? | false                    |
| CUDA Version    | N/A                      |
| cuDNN Version   | N/A                      |

I do not have conda environment available, so I cannot really help debugging further. Maybe it helps to completely re-create the environment? 

P.S.: If you are working with Google Colab or Kaggle, the following thread might be relevant: https://github.com/pykeen/pykeen/issues/373, as well as the notes in https://pykeen.readthedocs.io/en/stable/installation.html



Hello, I hit the same error when using 1.7.1-dev. All is good with 1.7.0.
Envs. for reference:
| Key             | Value                             |
|-----------------|-----------------------------------|
| OS              | posix                             |
| Platform        | Linux                             |
| Release         | 5.10.16.3-microsoft-standard-WSL2 |
| Time            | Tue Mar  8 12:23:09 2022          |
| Python          | 3.8.10                            |
| PyKEEN          | 1.7.1-dev                         |
| PyKEEN Hash     | UNHASHED                          |
| PyKEEN Branch   |                                   |
| PyTorch         | 1.10.2+cu102                      |
| CUDA Available? | true                              |
| CUDA Version    | 10.2                              |
| cuDNN Version   | 7605                              |



| Key             | Value                    |
|-----------------|--------------------------|
| OS              | posix                    |
| Platform        | Linux                    |
| Release         | 5.4.0-97-generic         |
| Time            | Tue Mar  8 11:28:57 2022 |
| Python          | 3.7.11                   |
| PyKEEN          | 1.7.1-dev                   |
| PyKEEN Hash     | UNHASHED                 |
| PyKEEN Branch   |                          |
| PyTorch         | 1.10.2                   |
| CUDA Available? | true                     |
| CUDA Version    | 11.3                     |
| cuDNN Version   | 8200                     |



I updated from 1.7.0 to 1.7.1 because with 1.7.0 I get the following error:

```python-traceback
Traceback (most recent call last):
  File "Train_KG.py", line 5, in <module>
    from utils.kge_train_utils import *
  File "/utils/kge_train_utils.py", line 7, in <module>
    from pykeen.pipeline import pipeline
  File "/envs/my_env/lib/python3.8/site-packages/pykeen/pipeline/__init__.py", line 5, in <module>
    from .api import (
  File "/envs/my_env/lib/python3.8/site-packages/pykeen/pipeline/api.py", line 205, in <module>
    from ..evaluation import Evaluator, MetricResults, evaluator_resolver
  File "/envs/my_env/lib/python3.8/site-packages/pykeen/evaluation/__init__.py", line 10, in <module>
    from .classification_evaluator import ClassificationEvaluator, ClassificationMetricResults
  File "/envs/my_env/lib/python3.8/site-packages/pykeen/evaluation/classification_evaluator.py", line 12, in <module>
    from .evaluator import Evaluator, MetricResults
  File "/envs/my_env/lib/python3.8/site-packages/pykeen/evaluation/evaluator.py", line 20, in <module>
    from ..models import Model
  File "/envs/my_env/lib/python3.8/site-packages/pykeen/models/__init__.py", line 17, in <module>
    from .unimodal import (
  File "/envs/my_env/lib/python3.8/site-packages/pykeen/models/unimodal/__init__.py", line 100, in <module>
    from .rgcn import RGCN
  File "/envs/my_env/lib/python3.8/site-packages/pykeen/models/unimodal/rgcn.py", line 13, in <module>
    from ...nn.message_passing import Decomposition, RGCNRepresentations
  File "/envs/my_env/lib/python3.8/site-packages/pykeen/nn/message_passing.py", line 11, in <module>
    from class_resolver.api import Hint
ImportError: cannot import name 'Hint' from 'class_resolver.api' (/envs/my_env/lib/python3.8/site-packages/class_resolver/api.py)
```

You can downgrade class-resolver to 0.3.3 to get 1.7.0 working again

Okay, I think I got the difference: In my test runs I did not run a GPU machine, and thus, no automatic memory optimization was happening. When I run the snippet on a GPU server, I encounter the same issue during the size probing.

> You can downgrade class-resolver to 0.3.3 to get 1.7.0 working again

Thank you, now it's working!



> Okay, I think I got the difference: In my test runs I did not run a GPU machine, and thus, no automatic memory optimization was happening. When I run the snippet on a GPU server, I encounter the same issue during the size probing.

Seems you are right - I have no issues with GPU disabled.