Repository: pykeen/pykeen
Issue: Performance regression for negative sampling (cf. #417)
Root cause: Algorithmic Error

### What is your question

Would it be possible to get some insights/info about the benchmarks discussed in #417 and how the different datasets/settings were affected?

I have noticed some big runtime increases after this change. 15 min. experiments turning into 5 hour adventures!

 

### Environment

| Key             | Value                    |
|-----------------|--------------------------|
| OS              | posix                    |
| Platform        | Linux                    |
| Release         | 5.4.0-97-generic         |
| Time            | Wed Mar  9 18:54:24 2022 |
| Python          | 3.7.11                   |
| PyKEEN          | 1.7.1-dev                   |
| PyKEEN Hash     | UNHASHED                 |
| PyKEEN Branch   |                          |
| PyTorch         | 1.10.2                   |
| CUDA Available? | true                     |
| CUDA Version    | 11.3                     |
| cuDNN Version   | 8200                     |

### Issue Template Checks

- [X] This is not a bug report (use a different issue template if it is)
- [X] This is not a feature request (use a different issue template if it is)
- [X] I've read the text explaining why including environment information is important and understand if I omit this information that my issue will be dismissed

They're in https://github.com/pykeen/collation-benchmark. This seemed to show that it didn't make a difference, but we also happened to notice the performance regression recently too

Thank you! Closing this issue, but I will update you if I find something regarding the performance

@Koenkalle Can you provide a setting (preferably including one of the pre-packaged datasets) where you experienced the run-time increase?

@mberr I have a noticeable difference in runtime between bcda4d2cdf327044b73e1b3aa6dd4d0cf6d16061 and  99a4ab83af98600c7414ba552486ff0e7e449e41 on this small example:
```
from pykeen.pipeline import pipeline
pipeline_result = pipeline(
                dataset="wn18",
                model="simple",
                model_kwargs=dict(
                    embedding_dim=200),
            )
```

I made an upscaled version to make the difference more clear:
```
from pykeen.pipeline import pipeline
pipeline_result = pipeline(
                dataset="wn18",
                model="simple",
                training_kwargs=dict(batch_size=1415),
                optimizer_kwargs=dict(lr=0.1),
                epochs=1000,
                regularizer_kwargs=dict(normalize=False,p=2.0, weight=0.03),
                negative_sampler='basic',
                negative_sampler_kwargs=dict(num_negs_per_pos=1, filtered=False, filterer='python-set')
)
```

Just checked the CI logs and it's definitely the case that the time jumped up on #417
<img width="963" alt="Screen Shot 2022-03-10 at 00 00 16" src="https://user-images.githubusercontent.com/5069736/157553489-c831695e-8fba-42f5-bd62-77f3147839cf.png">



I re-opened the issue, so we can use it to discuss the performance regression.

I ran the following code snippet
```python
from timeit import default_timer
from pykeen.pipeline import pipeline
start = default_timer()
pipeline(
        dataset="wn18",
        model="distmult",
        model_kwargs=dict(
            embedding_dim=200,
        ),
        training_kwargs=dict(
            num_epochs=1,
        ),
)
print("Training took", default_timer() - start ,"seconds")
```

for f04bd4f8201e2fc861a83a2a141df29271cd48ac (`master`), bcda4d2cdf327044b73e1b3aa6dd4d0cf6d16061 (just before #417 was merged), and 11dcb42a836534ecd3cfd83a1866d050f11ca231 (`batch-data-loader`, cf. #817 ) .

| commit | epoch time (negs=1) [sec] | epoch time (negs=256) [sec] |
| -- | --: | --: |
| f04bd4f8201e2fc861a83a2a141df29271cd48ac | 24.07 | 23.86 |
| 11dcb42a836534ecd3cfd83a1866d050f11ca231 | 9.15 | 18.55 |
| bcda4d2cdf327044b73e1b3aa6dd4d0cf6d16061 | 2.22 | 5.43 |

`num_workers=8`

| commit | epoch time (negs=1) [sec] | epoch time (negs=256) [sec] |
| -- | --: | --: |
| f04bd4f8201e2fc861a83a2a141df29271cd48ac | 4.45 | 4.24 |
| 11dcb42a836534ecd3cfd83a1866d050f11ca231 | 3.05 | 6.24 |
| bcda4d2cdf327044b73e1b3aa6dd4d0cf6d16061 | 3.16 | 6.13 |


<details>

Benchmarking Script
```bash
#!/usr/bin/env bash
git pull
git checkout master
python3 mwe.py
git checkout batch-data-loader
python3 mwe.py
git checkout bcda4d2cdf327044b73e1b3aa6dd4d0cf6d16061
python3 mwe.py
git checkout master
```

Full Output
```console
(venv) berrendorf@server:/tmp/pykeen$ ./bench.sh | tee res.txt
Already up to date.
Already on 'master'
Your branch is up to date with 'origin/master'.
No random seed is specified. Setting to 2949843806.
/tmp/pykeen/src/pykeen/nn/representation.py:361: UserWarning: Directly use Embedding.max_id instead of num_embeddings.
  warnings.warn(f"Directly use {self.__class__.__name__}.max_id instead of num_embeddings.")
/tmp/pykeen/src/pykeen/nn/representation.py:367: UserWarning: Directly use Embedding.shape instead of num_embeddings.
  warnings.warn(f"Directly use {self.__class__.__name__}.shape instead of num_embeddings.")
Training epochs on cuda:0: 100% [] 1/1 [00:24<00:00, 24.07s/epoch, loss=0.00392, prev_loss=nan]
INFO:pykeen.evaluation.evaluator:Starting batch_size search for evaluation now...                                                                                                  
INFO:pykeen.evaluation.evaluator:Concluded batch_size search with batch_size=128.
Evaluating on cuda:0: 100% [] 5.00k/5.00k [00:02<00:00, 1.90ktriple/s]
INFO:pykeen.evaluation.evaluator:Evaluation took 2.99s seconds
Training took 31.684266787953675 seconds
Switched to a new branch 'batch-data-loader'
Branch 'batch-data-loader' set up to track remote branch 'batch-data-loader' from 'origin'.
No random seed is specified. Setting to 399482701.
/tmp/pykeen/src/pykeen/nn/representation.py:361: UserWarning: Directly use Embedding.max_id instead of num_embeddings.
  warnings.warn(f"Directly use {self.__class__.__name__}.max_id instead of num_embeddings.")
/tmp/pykeen/src/pykeen/nn/representation.py:367: UserWarning: Directly use Embedding.shape instead of num_embeddings.
  warnings.warn(f"Directly use {self.__class__.__name__}.shape instead of num_embeddings.")
Training epochs on cuda:0: 100% [] 1/1 [00:09<00:00,  9.15s/epoch, loss=0.00392, prev_loss=nan]
INFO:pykeen.evaluation.evaluator:Starting batch_size search for evaluation now...                                                                                                  
INFO:pykeen.evaluation.evaluator:Concluded batch_size search with batch_size=128.
Evaluating on cuda:0: 100%[] 5.00k/5.00k [00:02<00:00, 1.90ktriple/s]
INFO:pykeen.evaluation.evaluator:Evaluation took 2.98s seconds
Training took 16.741263268049806 seconds
Note: switching to 'bcda4d2cdf327044b73e1b3aa6dd4d0cf6d16061'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at bcda4d2c 📥 🏭 Save Training Triples Factory  (#655)
No random seed is specified. Setting to 826507121.
Training epochs on cuda:0: 100% [] 1/1 [00:02<00:00,  2.22s/epoch, loss=0.00392, prev_loss=nan]
INFO:pykeen.evaluation.evaluator:Starting batch_size search for evaluation now...                                                                                                  
INFO:pykeen.evaluation.evaluator:Concluded batch_size search with batch_size=128.
Evaluating on cuda:0: 100%[] 5.00k/5.00k [00:02<00:00, 1.91ktriple/s]
INFO:pykeen.evaluation.evaluator:Evaluation took 2.96s seconds
Training took 9.70431349100545 seconds
Previous HEAD position was bcda4d2c 📥 🏭 Save Training Triples Factory  (#655)
Switched to branch 'master'
Your branch is up to date with 'origin/master'.
```
</details>

@Koenkalle could you check what happens if you increase the number of workers, by passing `num_workers=X`  (for 0 < X < number of cpu cores) as `training_kwargs`?

Using your setup:
| commit                                   | epoch time (negs=1) [sec] | epoch time (negs=256) [sec] |
|------------------------------------------|---------------------------|-----------------------------|
| f04bd4f8201e2fc861a83a2a141df29271cd48ac | 24.88                     | 26.79                       |
| 11dcb42a836534ecd3cfd83a1866d050f11ca231 | 14.99                     | 11.05                       |
| bcda4d2cdf327044b73e1b3aa6dd4d0cf6d16061 | 8.59                      | 11.06                       |

num_workers=8
| commit                                   | epoch time (negs=1) [sec] | epoch time (negs=256) [sec] |
|------------------------------------------|---------------------------|-----------------------------|
| f04bd4f8201e2fc861a83a2a141df29271cd48ac | 9.28                      | 9.29                        |
| 11dcb42a836534ecd3cfd83a1866d050f11ca231 | 23.51                     | 41.00                       |
| bcda4d2cdf327044b73e1b3aa6dd4d0cf6d16061 | 9.21                      | 10.91                       |



I get the following warning A LOT on 11dcb42a836534ecd3cfd83a1866d050f11ca231 when increasing the number of workers.
>` [...]/torch/utils/data/dataloader.py:533: UserWarning: Length of IterableDataset <src.pykeen.triples.instances.BatchedSLCWAInstances object at 0x7f2584eb5ed0> was reported to be 553 (when accessing len(dataloader)), but 6012 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset for examples.`



> I get the following warning A LOT on https://github.com/pykeen/pykeen/commit/11dcb42a836534ecd3cfd83a1866d050f11ca231 when increasing the number of workers.

should be fixed with 169a35b15a511312b2f9d442047df607ee96c905

New tables with fix from https://github.com/pykeen/pykeen/commit/169a35b15a511312b2f9d442047df607ee96c905:
`num_workers=1`
| commit                                   | epoch time (negs=1) [sec] | epoch time (negs=256) [sec] |
|------------------------------------------|---------------------------|-----------------------------|
| f04bd4f8201e2fc861a83a2a141df29271cd48ac | 24.56                     | 24.76                       |
| 169a35b15a511312b2f9d442047df607ee96c905 | 8.76                      | 10.66                       |
| bcda4d2cdf327044b73e1b3aa6dd4d0cf6d16061 | 8.73                      | 10.75                       |


`num_workers=8`
| commit                                   | epoch time (negs=1) [sec] | epoch time (negs=256) [sec] |
|------------------------------------------|---------------------------|-----------------------------|
| f04bd4f8201e2fc861a83a2a141df29271cd48ac | 9.42                      | 9.46                        |
| 169a35b15a511312b2f9d442047df607ee96c905 | 9.19                      | 11.12                       |
| bcda4d2cdf327044b73e1b3aa6dd4d0cf6d16061 | 8.85                      | 10.95                       |

`num_workers=24`
| commit                                   | epoch time (negs=1) [sec] | epoch time (negs=256) [sec] |
|------------------------------------------|---------------------------|-----------------------------|
| f04bd4f8201e2fc861a83a2a141df29271cd48ac | 9.43                      | 9.36                        |
| 169a35b15a511312b2f9d442047df607ee96c905 | 9.43                      | 11.66                       |
| bcda4d2cdf327044b73e1b3aa6dd4d0cf6d16061 | 9.73                      | 11.39                       |


@Koenkalle if you want to join our PyKEEN slack, please shoot me an email at cthoyt@gmail.com

Closed by #817