Repository: pykeen/pykeen
Issue: Basic Negative Sampler issue on some datasets with specific models
Root cause: Validation Gap


[dataset1.txt](https://github.com/pykeen/pykeen/files/6414058/dataset1.txt)
[dataset2.txt](https://github.com/pykeen/pykeen/files/6414059/dataset2.txt)

Hi, I'm in a internship where I work on the impact of noisy datasets where I need to get Embeddings to compare them.

The bug happened to me on 2 corruptions of datasets, I got this error:
```
pline = pipeline(training=training_file, testing=training_file, model=model, training_kwargs=dict(num_epochs=epochs), random_seed=random_seed)
  File "/home/leyto/.local/share/virtualenvs/kge-fDonqShT/lib/python3.9/site-packages/pykeen/pipeline/api.py", line 977, in pipeline
    losses = training_loop_instance.train(
  File "/home/leyto/.local/share/virtualenvs/kge-fDonqShT/lib/python3.9/site-packages/pykeen/training/training_loop.py", line 289, in train
    result = self._train(
  File "/home/leyto/.local/share/virtualenvs/kge-fDonqShT/lib/python3.9/site-packages/pykeen/training/training_loop.py", line 545, in _train
    current_epoch_loss += self._forward_pass(
  File "/home/leyto/.local/share/virtualenvs/kge-fDonqShT/lib/python3.9/site-packages/pykeen/training/training_loop.py", line 629, in _forward_pass
    loss = self._process_batch(
  File "/home/leyto/.local/share/virtualenvs/kge-fDonqShT/lib/python3.9/site-packages/pykeen/training/slcwa.py", line 96, in _process_batch
    neg_samples, neg_samples_filter = self.negative_sampler.sample(positive_batch=positive_batch)
  File "/home/leyto/.local/share/virtualenvs/kge-fDonqShT/lib/python3.9/site-packages/pykeen/sampling/basic_negative_sampler.py", line 84, in sample
    for index, start in zip(self._corruption_indices, range(0, num_negs, split_idx)):
ValueError: range() arg 3 must not be zero
```
This bug only occurs on 2 corrupted dataset (on a total of 450 corrupted datasets) on at least those 3 Models:
- TransE
- RotatE
- ComplEx
- SimplE
But worked fine with at least ERMLPE on the same data and the same random_seed (even if changing it didn't solve the issue)


**To Reproduce**
```python
pline = pipeline(training=training_file, testing=training_file, model=model, training_kwargs=dict(num_epochs=epochs), random_seed=42)
```
Where training_file is one of the 2 datasets, model one of the four model (TransE, RotatE, ComplEx, SimplE, or maybe some other) and epochs 5 for example but the problem isn't at this level.


**Environment:**
| Key                  | Value             |
|----------------------|-------------------|
| `os.name`            | posix             |
| `platform.system()`  | Linux             |
| `platform.release()` | 5.4.114-1-MANJARO |
| python               | 3.9.4             |
| pykeen               | 1.4.0-UNHASHED    |
| torch                | 1.8.1+cu102       |
| cuda available       | false             |
| cuda                 | 10.2              |
| cudnn                | 7605              |

I don't have GPU to perform Embeddings.

**Additional information**
I know the testing is the training, I perform the training on the full dataset to get the trained Embeddings. Even reducing the file passed to testing didn't change the issue.

The first dataset is a corrupted version of NATIONS where the entity netherland got duplicated into a new entity named netherland__DUP1 and have the same relations that the original one (which still remain).
The second one is a corrupted version of UMLS where some entities got merged into entity with name NULL keeping their previous relations.

Thanks for your help, your work is great !


Hi @DylanVsn .

thanks for reporting this issue! The problem arises since the last batch contains fewer triples than there are corruption indices (2, head & tail). I pushed a fix in #415.

Hi @mberr Thanks for the quick fix, it solves the issue !

(I know it's a bit out of context but this fix come with a new advanced version of pykeen I didn't got with pypi and now I was wondering which was the way to get a pair "entity_name": embedding and the same goes for the relations because my previous
```python
keys = list(pline.model.triples_factory.entity_to_id.keys())
values = pline.model.entity_embeddings()[pline.model.triples_factory.entities_to_ids(keys)].tolist()
nodes_dict = dict(zip(keys, values))
```
isn't valid anymore due to the removing of triples_factory from the pline.model.)

Thanks again for your help.

@DylanVsn the triples factory is not part of the model anymore. However, it is still part of the training result.

Thus, you can modify your code as follows

```python
x = result.model.entity_embeddings().tolist()
nodes_dict = {
    label: x[i]
    for label, i in result.training.entity_to_id.items()
}
```

P.S.: For new-style models, such as e.g., Unstructured Model, or PairRE, you need to replace the first line by
```python
x = result.model.entity_representations[0]().tolist()
```

More information on how to use the "learned embeddings" under this header in the docs: https://pykeen.readthedocs.io/en/latest/tutorial/first_steps.html#using-learned-embeddings

Please let me know if some of that isn't clear and I will make appropriate updates

Thanks to both of you @mberr and @cthoyt for your replies it was exactly what I was looking for.
Thank you for this great package, keep up the good work !

@DylanVsn I'd be interested to know more about your algorithm for "corruption".

I've investigating the impact of random initialization and dataset split (there is a section about halfway down this blog post on it https://cthoyt.com/2021/04/05/kgem-perspectives.html) as well as some experiments I haven't published yet on randomly robustness to removing triples from the training sets. The implementation for resplitting the dataset is in [pykeen.datasets.base.Dataset.remix](https://pykeen.readthedocs.io/en/latest/api/pykeen.datasets.base.Dataset.html?highlight=remix#pykeen.datasets.base.Dataset.remix) and removing triples is in [pykeen.datasets.base.Dataset.deteriorate](https://pykeen.readthedocs.io/en/latest/api/pykeen.datasets.base.Dataset.html?highlight=remix#pykeen.datasets.base.Dataset.deteriorate).

Hi @cthoyt thanks for your interesting blog post, that was an interesting and rich reading.
Thank you again for the two methods of PyKeen, it could be usefull in near future.

Concerning the algorithms we currently have 3 algorithms that work on the graph instead of the triples and could be on either entities or relations. For each corruption, we specify if it either affects entities or relations and the percentage to corrupt. (i.e. we want to corrupt 30% of entities, 30% of all distinct entities will be marked for corruption, in case of relations, a set of relation's name is made and we select 30% of the names; all relations that have its name selected will be corrupted.) We are aware that for relations the corruptions will be unbalanced if the relations distribution is unbalanced itself (to be balanced here we must use triples view instead of the graph one). Concerning the current corruptions we have:
- Nullification: change the name of the object to "NULL" to simulate missing facts.
- Typographical error: change the name of the object by adding a mark at the end of it (work better on triples because on graph view it changes only the name and the id it gets in KGEM will be the same and nothing will be changed)
- Duplication: duplicate the object, in case of an entity, it will copy all the relations of the original entity too.

We will study the impact of the corruption on the learned Embeddings first, especially on those which changed without being corrupted, in a second part of the study we will compare scores of KGEMs on some corruptions to find out if some KGEMs perform better against corrupted/noisy datasets.