Repository: pykeen/pykeen
Issue: ValueError: Ambiguous device for some reproducibility configurations
Root cause: Validation Gap

### Describe the bug

Running, e.g., `pykeen experiments reproduce quate zhang2019 fb15k237` on master (73edddf747987f23c22fa5242a5569363282d7fd) on GPU results in a ValueError due to mixed devices. This seems to be caused by 

https://github.com/pykeen/pykeen/blob/73edddf747987f23c22fa5242a5569363282d7fd/src/pykeen/training/training_loop.py#L520-L521

where

https://github.com/pykeen/pykeen/blob/73edddf747987f23c22fa5242a5569363282d7fd/src/pykeen/training/training_loop.py#L149-L152

### How to reproduce

```console
$ pykeen experiments reproduce quate zhang2019 fb15k237
Unable to handle parameter in AutoSF: coefficients
Running configuration at ../src/pykeen/experiments/quate/zhang2019_quate_fb15k237.json
2022-02-05 11:20:28 INFO     Running: Learn FB15K-237 Data Set with QuatE^1 as described by Zhang et al., 2019.
2022-02-05 11:20:28 WARNING  No random seed is specified. Setting to 728816410.
2022-02-05 11:20:28 INFO     Using device: None
2022-02-05 11:20:29 WARNING  You're trying to map triples with 30 entities and 0 relations that are not in the training set. These triples will be excluded from the mapping.
2022-02-05 11:20:29 WARNING  In total 28 from 20466 triples were filtered out
2022-02-05 11:20:29 WARNING  You're trying to map triples with 9 entities and 0 relations that are not in the training set. These triples will be excluded from the mapping.
2022-02-05 11:20:29 WARNING  In total 9 from 17535 triples were filtered out
Traceback (most recent call last):
  File "../venv/bin/pykeen", line 33, in <module>
    sys.exit(load_entry_point('pykeen', 'console_scripts', 'pykeen')())
  File "../venv/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "../venv/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "../venv/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "../venv/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "../venv/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "../venv/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "../src/pykeen/experiments/cli.py", line 105, in reproduce
    _help_reproduce(
  File "../src/pykeen/experiments/cli.py", line 185, in _help_reproduce
    replicate_pipeline_from_path(
  File "../src/pykeen/pipeline/api.py", line 495, in replicate_pipeline_from_path
    replicate_pipeline_from_config(config=load_configuration(path), **kwargs)
  File "../src/pykeen/pipeline/api.py", line 516, in replicate_pipeline_from_config
    save_pipeline_results_to_directory(
  File "../src/pykeen/pipeline/api.py", line 649, in save_pipeline_results_to_directory
    for i, pipeline_result in enumerate(pipeline_results):
  File "../src/pykeen/pipeline/api.py", line 515, in <genexpr>
    pipeline_results = (pipeline_from_config(config, **kwargs) for _ in range(replicates))
  File "../src/pykeen/pipeline/api.py", line 710, in pipeline_from_config
    return pipeline(
  File "../src/pykeen/pipeline/api.py", line 1217, in pipeline
    losses = training_loop_instance.train(
  File "../src/pykeen/training/training_loop.py", line 337, in train
    result = self._train(
  File "../src/pykeen/training/training_loop.py", line 478, in _train
    sub_batch_size, slice_size = self.sub_batch_and_slice(
  File "../src/pykeen/training/training_loop.py", line 907, in sub_batch_and_slice
    sub_batch_size, finished_search, supports_sub_batching = self._sub_batch_size_search(
  File "../src/pykeen/training/training_loop.py", line 986, in _sub_batch_size_search
    self._train(
  File "../src/pykeen/training/training_loop.py", line 521, in _train
    self.model = self.model.to(self.device)
  File "../src/pykeen/training/training_loop.py", line 152, in device
    return self.model.device
  File "../src/pykeen/models/base.py", line 147, in device
    raise ValueError(f"Ambiguous device! Found: {devices}\n\n{info}")
ValueError: Ambiguous device! Found: {device(type='cuda', index=0), device(type='cpu')}

{device(type='cpu'): ['entity_representations.0._embeddings.weight', 'relation_representations.0._embeddings.weight'], device(type='cuda', index=0): ['entity_representations.0.regularizer.regularization_term', 'entity_representations.0.regularizer.weight', 'relation_representations.0.regularizer.regularization_term', 'relation_representations.0.regularizer.weight']}
```

### Environment

| Key             | Value                    |
|-----------------|--------------------------|
| OS              | posix                    |
| Platform        | Linux                    |
| Release         | 5.4.0-77-generic         |
| Time            | Sat Feb  5 11:29:42 2022 |
| Python          | 3.8.5                    |
| PyKEEN          | 1.7.1-dev                |
| PyKEEN Hash     | 73edddf7                 |
| PyKEEN Branch   | master                   |
| PyTorch         | 1.10.0+cu113             |
| CUDA Available? | true                     |
| CUDA Version    | 11.3                     |
| cuDNN Version   | 8200                     |

### Additional information

A simple fix would be the send the model to the selected device before going into `self._train`
https://github.com/pykeen/pykeen/blob/73edddf747987f23c22fa5242a5569363282d7fd/src/pykeen/training/training_loop.py#L337