Repository: pykeen/pykeen
Issue: RuntimeError: CUDA error: device-side assert triggered
Root cause: Validation Gap

**Hi**
[I got a `RuntimeError` when I tried to train a model on my own data with inverse triples.]

**Here are my codes.**
```python
# dataset preparation
tf = TriplesFactory(path='data.tsv')  
train_valid,test = tf.split([.9,.1],random_state=0)
train,valid = train_valid.split([.95,.05],random_state=0)

pd.DataFrame(train.triples).to_csv('train.tsv',index=False,header=None,sep='\t')
pd.DataFrame(valid.triples).to_csv('valid.tsv',index=False,header=None,sep='\t')
pd.DataFrame(test.triples).to_csv('test.tsv',index=False,header=None,sep='\t')
train = TriplesFactory(path='train.tsv',create_inverse_triples=True) 
valid = TriplesFactory(path='valid.tsv',create_inverse_triples=True,    
                    entity_to_id=train.entity_to_id,
                    relation_to_id=train.relation_to_id,) 
test = TriplesFactory(path='test.tsv',create_inverse_triples=True,
                    entity_to_id=train.entity_to_id,
                    relation_to_id=train.relation_to_id,) 

# model train
pipeline_result = pipeline(
    training=train,
    testing=test,
    validation=valid,
    model='TransE',
    stopper='early',
    loss='NSSALoss',
    model_kwargs=dict(embedding_dim=400,automatic_memory_optimization=True),
    stopper_kwargs=dict(frequency=5, patience=3, relative_delta=0.002),
    training_kwargs=dict(num_epochs=epoch),
    evaluation_kwargs=dict(do_time_consuming_checks=False,use_tqdm=False),
    random_seed=0,
    device='cuda:2'
)
```

**Error message**
```python-traceback
RuntimeError                              Traceback (most recent call last)
/data/hadmap-big/nicoleqye/kge/lib/python3.7/site-packages/pykeen/evaluation/evaluator.py in _param_size_search(self, key, start_value, model, mapped_triples, device, use_tqdm, restrict_entities_to, do_time_consuming_checks)
    284                     restrict_entities_to=restrict_entities_to,
--> 285                     do_time_consuming_checks=do_time_consuming_checks,
    286                 )

/data/hadmap-big/nicoleqye/kge/lib/python3.7/site-packages/pykeen/evaluation/evaluator.py in evaluate(model, mapped_triples, evaluators, only_size_probing, batch_size, slice_size, device, squeeze, use_tqdm, tqdm_kwargs, restrict_entities_to, do_time_consuming_checks)
    568                     positive_masks_required=positive_masks_required,
--> 569                     filtering_necessary=filtering_necessary,
    570                 )

/data/hadmap-big/nicoleqye/kge/lib/python3.7/site-packages/pykeen/evaluation/evaluator.py in _evaluate_batch(batch, model, column, filtered_evaluators, unfiltered_evaluators, slice_size, all_pos_triples, relation_filter, restrict_entities_to, positive_masks_required, filtering_necessary)
    649         torch.arange(0, batch.shape[0]),
--> 650         batch[:, column],
    651     ]

RuntimeError: CUDA error: device-side assert triggered
```

I guess the problem happened in the evaluation stage because the training completed 4 epochs successfully.  
`Training epochs on cuda:2:   0%|▏                                                | 4/1000 [00:17<1:11:41,  4.32s/epoch, loss=0.00389, prev_loss=0.00541]`

**I don't know how to fix it. Is there anything wrong in my code?**



Hey @yeqing97 thats quite an interesting error! I think this is something for @mberr or @lvermue to look into

Hi @yeqing97 ,

`RuntimeError: CUDA error: device-side assert triggered` is usually raised when an index out of bounds errors occurs on CUDA device. Before the Python error log you often see `ATen` error messages such as described in [this PyTorch issue](https://github.com/pytorch/pytorch/issues/48530).

Could you please check for additional error messages above this? Also, running on cpu may yield better error messages.

**Thanks for your suggestion!**
Here are my test results.

<summary><strong>1. training on GPU</strong></summary>

<details>

```python-traceback
In [8]: pipeline_result = pipeline(
   ...:     training=train,
   ...:     testing=test,
   ...:     validation=valid,
   ...:     model='TransE',
   ...:     stopper='early',
   ...:     loss='NSSALoss',
   ...:     model_kwargs=dict(embedding_dim=400,automatic_memory_optimization=True),
   ...:     stopper_kwargs=dict(frequency=5, patience=3, relative_delta=0.002),
   ...:     training_kwargs=dict(num_epochs=1000),
   ...:     evaluation_kwargs=dict(do_time_consuming_checks=False,use_tqdm=False),
   ...:     random_seed=0,
   ...:     device='cuda:2'
   ...: )
Training epochs on cuda:2:   0%|▏                                                | 4/1000 [00:17<1:10:56,  4.27s/epoch, loss=0.00389, prev_loss=0.00541]
---------------------------------------------------------------------------                                                                             
RuntimeError                              Traceback (most recent call last)
/data/hadmap-big/nicoleqye/kge/lib/python3.7/site-packages/pykeen/evaluation/evaluator.py in _param_size_search(self, key, start_value, model, mapped_triples, device, use_tqdm, restrict_entities_to, do_time_consuming_checks)
    284                     restrict_entities_to=restrict_entities_to,
--> 285                     do_time_consuming_checks=do_time_consuming_checks,
    286                 )

/data/hadmap-big/nicoleqye/kge/lib/python3.7/site-packages/pykeen/evaluation/evaluator.py in evaluate(model, mapped_triples, evaluators, only_size_probing, batch_size, slice_size, device, squeeze, use_tqdm, tqdm_kwargs, restrict_entities_to, do_time_consuming_checks)
    568                     positive_masks_required=positive_masks_required,
--> 569                     filtering_necessary=filtering_necessary,
    570                 )

/data/hadmap-big/nicoleqye/kge/lib/python3.7/site-packages/pykeen/evaluation/evaluator.py in _evaluate_batch(batch, model, column, filtered_evaluators, unfiltered_evaluators, slice_size, all_pos_triples, relation_filter, restrict_entities_to, positive_masks_required, filtering_necessary)
    644     else:
--> 645         batch_scores_of_corrupted = model.predict_scores_all_heads(batch[:, 1:3], slice_size=slice_size)
    646 

/data/hadmap-big/nicoleqye/kge/lib/python3.7/site-packages/pykeen/models/base.py in predict_scores_all_heads(self, rt_batch, slice_size)
    606         if slice_size is None:
--> 607             scores = self.score_t(rt_batch_cloned)
    608         else:

/data/hadmap-big/nicoleqye/kge/lib/python3.7/site-packages/pykeen/models/unimodal/trans_e.py in score_t(self, hr_batch)
    108 
--> 109         return -torch.norm(h[:, None, :] + r[:, None, :] - t[None, :, :], dim=-1, p=self.scoring_fct_norm)
    110 

RuntimeError: CUDA out of memory. Tried to allocate 9.73 GiB (GPU 2; 15.75 GiB total capacity; 162.40 MiB already allocated; 2.76 GiB free; 182.00 MiB reserved in total by PyTorch)

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
<ipython-input-8-3d6ff473536c> in <module>
     11     evaluation_kwargs=dict(do_time_consuming_checks=False,use_tqdm=False),
     12     random_seed=0,
---> 13     device='cuda:2'
     14 )

/data/hadmap-big/nicoleqye/kge/lib/python3.7/site-packages/pykeen/pipeline.py in pipeline(dataset, dataset_kwargs, training, testing, validation, evaluation_entity_whitelist, evaluation_relation_whitelist, model, model_kwargs, loss, loss_kwargs, regularizer, regularizer_kwargs, optimizer, optimizer_kwargs, clear_optimizer, training_loop, negative_sampler, negative_sampler_kwargs, training_kwargs, stopper, stopper_kwargs, evaluator, evaluator_kwargs, evaluation_kwargs, result_tracker, result_tracker_kwargs, metadata, device, random_seed, use_testing_data)
   1003         result_tracker=result_tracker,
   1004         clear_optimizer=clear_optimizer,
-> 1005         **training_kwargs,
   1006     )
   1007     training_end_time = time.time() - training_start_time

/data/hadmap-big/nicoleqye/kge/lib/python3.7/site-packages/pykeen/training/training_loop.py in train(self, num_epochs, batch_size, slice_size, label_smoothing, sampler, continue_training, only_size_probing, use_tqdm, use_tqdm_batch, tqdm_kwargs, stopper, result_tracker, sub_batch_size, num_workers, clear_optimizer)
    194             result_tracker=result_tracker,
    195             sub_batch_size=sub_batch_size,
--> 196             num_workers=num_workers,
    197         )
    198 

/data/hadmap-big/nicoleqye/kge/lib/python3.7/site-packages/pykeen/training/training_loop.py in _train(self, num_epochs, batch_size, slice_size, label_smoothing, sampler, continue_training, only_size_probing, use_tqdm, use_tqdm_batch, tqdm_kwargs, stopper, result_tracker, sub_batch_size, num_workers)
    413             })
    414 
--> 415             if stopper is not None and stopper.should_evaluate(epoch) and stopper.should_stop(epoch):
    416                 return self.losses_per_epochs
    417 

/data/hadmap-big/nicoleqye/kge/lib/python3.7/site-packages/pykeen/stoppers/early_stopping.py in should_stop(self, epoch)
    132             slice_size=self.evaluation_slice_size,
    133             # Only perform time consuming checks for the first call.
--> 134             do_time_consuming_checks=self.evaluation_batch_size is None,
    135         )
    136         # After the first evaluation pass the optimal batch and slice size is obtained and saved for re-use

/data/hadmap-big/nicoleqye/kge/lib/python3.7/site-packages/pykeen/evaluation/evaluator.py in evaluate(self, model, mapped_triples, batch_size, slice_size, device, use_tqdm, tqdm_kwargs, restrict_entities_to, do_time_consuming_checks)
    144                 use_tqdm=False,
    145                 restrict_entities_to=restrict_entities_to,
--> 146                 do_time_consuming_checks=do_time_consuming_checks,
    147             )
    148             # The batch_size and slice_size should be accessible to outside objects for re-use, e.g. early stoppers.

/data/hadmap-big/nicoleqye/kge/lib/python3.7/site-packages/pykeen/evaluation/evaluator.py in batch_and_slice(self, model, mapped_triples, batch_size, device, use_tqdm, restrict_entities_to, do_time_consuming_checks)
    214             use_tqdm=use_tqdm,
    215             restrict_entities_to=restrict_entities_to,
--> 216             do_time_consuming_checks=do_time_consuming_checks,
    217         )
    218 

/data/hadmap-big/nicoleqye/kge/lib/python3.7/site-packages/pykeen/evaluation/evaluator.py in _param_size_search(self, key, start_value, model, mapped_triples, device, use_tqdm, restrict_entities_to, do_time_consuming_checks)
    293                 # The cache of the previous run has to be freed to allow accurate memory availability estimates
    294                 gc.collect()
--> 295                 torch.cuda.empty_cache()
    296                 if not is_cudnn_error(runtime_error) and not is_cuda_oom_error(runtime_error):
    297                     raise runtime_error

/data/hadmap-big/nicoleqye/kge/lib/python3.7/site-packages/torch/cuda/memory.py in empty_cache()
     33     """
     34     if is_initialized():
---> 35         torch._C._cuda_emptyCache()
     36 
     37 

RuntimeError: CUDA error: device-side assert triggered
````
</details>

<summary><strong>2. training on CPU</strong></summary>

<details>

```python-traceback
In [7]: pipeline_result = pipeline(
   ...:     training=train,
   ...:     testing=test,
   ...:     validation=valid,
   ...:     model='TransE',
   ...:     stopper='early',
   ...:     loss='NSSALoss',
   ...:     model_kwargs=dict(embedding_dim=400,automatic_memory_optimization=True),
   ...:     stopper_kwargs=dict(frequency=5, patience=3, relative_delta=0.002),
   ...:     training_kwargs=dict(num_epochs=1000),
   ...:     evaluation_kwargs=dict(do_time_consuming_checks=False,use_tqdm=False),
   ...:     random_seed=0,
   ...:     device='cpu'
   ...: )
Training epochs on cpu:   0%|▏                                                  | 4/1000 [09:21<38:49:58, 140.36s/epoch, loss=0.0038, prev_loss=0.00527]
---------------------------------------------------------------------------                                                                             
RuntimeError                              Traceback (most recent call last)
<ipython-input-7-aaff01956e1c> in <module>
     11     evaluation_kwargs=dict(do_time_consuming_checks=False,use_tqdm=False),
     12     random_seed=0,
---> 13     device='cpu'
     14 )

/data/hadmap-big/nicoleqye/kge/lib/python3.7/site-packages/pykeen/pipeline.py in pipeline(dataset, dataset_kwargs, training, testing, validation, evaluation_entity_whitelist, evaluation_relation_whitelist, model, model_kwargs, loss, loss_kwargs, regularizer, regularizer_kwargs, optimizer, optimizer_kwargs, clear_optimizer, training_loop, negative_sampler, negative_sampler_kwargs, training_kwargs, stopper, stopper_kwargs, evaluator, evaluator_kwargs, evaluation_kwargs, result_tracker, result_tracker_kwargs, metadata, device, random_seed, use_testing_data)
   1003         result_tracker=result_tracker,
   1004         clear_optimizer=clear_optimizer,
-> 1005         **training_kwargs,
   1006     )
   1007     training_end_time = time.time() - training_start_time

/data/hadmap-big/nicoleqye/kge/lib/python3.7/site-packages/pykeen/training/training_loop.py in train(self, num_epochs, batch_size, slice_size, label_smoothing, sampler, continue_training, only_size_probing, use_tqdm, use_tqdm_batch, tqdm_kwargs, stopper, result_tracker, sub_batch_size, num_workers, clear_optimizer)
    194             result_tracker=result_tracker,
    195             sub_batch_size=sub_batch_size,
--> 196             num_workers=num_workers,
    197         )
    198 

/data/hadmap-big/nicoleqye/kge/lib/python3.7/site-packages/pykeen/training/training_loop.py in _train(self, num_epochs, batch_size, slice_size, label_smoothing, sampler, continue_training, only_size_probing, use_tqdm, use_tqdm_batch, tqdm_kwargs, stopper, result_tracker, sub_batch_size, num_workers)
    413             })
    414 
--> 415             if stopper is not None and stopper.should_evaluate(epoch) and stopper.should_stop(epoch):
    416                 return self.losses_per_epochs
    417 

/data/hadmap-big/nicoleqye/kge/lib/python3.7/site-packages/pykeen/stoppers/early_stopping.py in should_stop(self, epoch)
    132             slice_size=self.evaluation_slice_size,
    133             # Only perform time consuming checks for the first call.
--> 134             do_time_consuming_checks=self.evaluation_batch_size is None,
    135         )
    136         # After the first evaluation pass the optimal batch and slice size is obtained and saved for re-use

/data/hadmap-big/nicoleqye/kge/lib/python3.7/site-packages/pykeen/evaluation/evaluator.py in evaluate(self, model, mapped_triples, batch_size, slice_size, device, use_tqdm, tqdm_kwargs, restrict_entities_to, do_time_consuming_checks)
    144                 use_tqdm=False,
    145                 restrict_entities_to=restrict_entities_to,
--> 146                 do_time_consuming_checks=do_time_consuming_checks,
    147             )
    148             # The batch_size and slice_size should be accessible to outside objects for re-use, e.g. early stoppers.

/data/hadmap-big/nicoleqye/kge/lib/python3.7/site-packages/pykeen/evaluation/evaluator.py in batch_and_slice(self, model, mapped_triples, batch_size, device, use_tqdm, restrict_entities_to, do_time_consuming_checks)
    214             use_tqdm=use_tqdm,
    215             restrict_entities_to=restrict_entities_to,
--> 216             do_time_consuming_checks=do_time_consuming_checks,
    217         )
    218 

/data/hadmap-big/nicoleqye/kge/lib/python3.7/site-packages/pykeen/evaluation/evaluator.py in _param_size_search(self, key, start_value, model, mapped_triples, device, use_tqdm, restrict_entities_to, do_time_consuming_checks)
    295                 torch.cuda.empty_cache()
    296                 if not is_cudnn_error(runtime_error) and not is_cuda_oom_error(runtime_error):
--> 297                     raise runtime_error
    298                 if values_dict[key] == 1:
    299                     logger.debug(

/data/hadmap-big/nicoleqye/kge/lib/python3.7/site-packages/pykeen/evaluation/evaluator.py in _param_size_search(self, key, start_value, model, mapped_triples, device, use_tqdm, restrict_entities_to, do_time_consuming_checks)
    283                     use_tqdm=use_tqdm,
    284                     restrict_entities_to=restrict_entities_to,
--> 285                     do_time_consuming_checks=do_time_consuming_checks,
    286                 )
    287                 evaluated_once = True

/data/hadmap-big/nicoleqye/kge/lib/python3.7/site-packages/pykeen/evaluation/evaluator.py in evaluate(model, mapped_triples, evaluators, only_size_probing, batch_size, slice_size, device, squeeze, use_tqdm, tqdm_kwargs, restrict_entities_to, do_time_consuming_checks)
    567                     restrict_entities_to=restrict_entities_to,
    568                     positive_masks_required=positive_masks_required,
--> 569                     filtering_necessary=filtering_necessary,
    570                 )
    571 

/data/hadmap-big/nicoleqye/kge/lib/python3.7/site-packages/pykeen/evaluation/evaluator.py in _evaluate_batch(batch, model, column, filtered_evaluators, unfiltered_evaluators, slice_size, all_pos_triples, relation_filter, restrict_entities_to, positive_masks_required, filtering_necessary)
    643         batch_scores_of_corrupted = model.predict_scores_all_tails(batch[:, 0:2], slice_size=slice_size)
    644     else:
--> 645         batch_scores_of_corrupted = model.predict_scores_all_heads(batch[:, 1:3], slice_size=slice_size)
    646 
    647     # Select scores of true

/data/hadmap-big/nicoleqye/kge/lib/python3.7/site-packages/pykeen/models/base.py in predict_scores_all_heads(self, rt_batch, slice_size)
    605         rt_batch_cloned = rt_batch_cloned.flip(1)
    606         if slice_size is None:
--> 607             scores = self.score_t(rt_batch_cloned)
    608         else:
    609             scores = self.score_t(rt_batch_cloned, slice_size=slice_size)

/data/hadmap-big/nicoleqye/kge/lib/python3.7/site-packages/pykeen/models/unimodal/trans_e.py in score_t(self, hr_batch)
    104         # Get embeddings
    105         h = self.entity_embeddings(hr_batch[:, 0])
--> 106         r = self.relation_embeddings(hr_batch[:, 1])
    107         t = self.entity_embeddings.weight
    108 

/data/hadmap-big/nicoleqye/kge/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

/data/hadmap-big/nicoleqye/kge/lib/python3.7/site-packages/torch/nn/modules/sparse.py in forward(self, input)
    112         return F.embedding(
    113             input, self.weight, self.padding_idx, self.max_norm,
--> 114             self.norm_type, self.scale_grad_by_freq, self.sparse)
    115 
    116     def extra_repr(self):

/data/hadmap-big/nicoleqye/kge/lib/python3.7/site-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1482         # remove once script supports set_grad_enabled
   1483         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1484     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1485 
   1486 

RuntimeError: index out of range: Tried to access index 1277920688 out of table with 1 rows. at /opt/conda/conda-bld/pytorch_1579040055865/work/aten/src/TH/generic/THTensorEvenMoreMath.cpp:418
```

</details>

Hi @yeqing97 .

From the failing index

`RuntimeError: index out of range: Tried to access index 1277920688 out of table with 1 rows.`

it looks like this is using an unitialized long array. Also it seems to be related to evaluation when using inverse triples.


When I run the following snippet on the current master ([c52787a3](https://github.com/pykeen/pykeen/tree/c52787a3f65e47f3135b3698864f10fdb6d78349)), I do not observe any error.

```python
from pykeen.pipeline import pipeline

pipeline_result = pipeline(
    dataset="nations",
    dataset_kwargs=dict(create_inverse_triples=True),
    model="TransE",
    stopper="early",
    loss="NSSALoss",
    model_kwargs=dict(embedding_dim=400),
    stopper_kwargs=dict(frequency=5, patience=3, relative_delta=0.002),
    training_kwargs=dict(num_epochs=1000),
    evaluation_kwargs=dict(do_time_consuming_checks=False, use_tqdm=False),
    random_seed=0,
    device="cpu",
    automatic_memory_optimization=True,
)
```

Could you please run the same code and see whether you observe an error?

Hi! @mberr 
I have checked the snippet you mentioned and it runs successfully.
```
In [3]: pipeline_result = pipeline(
   ...:     dataset="nations",
   ...:     dataset_kwargs=dict(create_inverse_triples=True),
   ...:     model="TransE",
   ...:     stopper="early",
   ...:     loss="NSSALoss",
   ...:     model_kwargs=dict(embedding_dim=400),
   ...:     stopper_kwargs=dict(frequency=5, patience=3, relative_delta=0.002),
   ...:     training_kwargs=dict(num_epochs=1000),
   ...:     evaluation_kwargs=dict(do_time_consuming_checks=False, use_tqdm=False),
   ...:     random_seed=0,
   ...:     device="cpu",)
Training epochs on cpu:   0%|▏                                                     | 4/1000 [00:01<04:07,  4.02epoch/s, loss=0.00372, prev_loss=0.00434]INFO:pykeen.evaluation.evaluator:Concluded batch_size search with batch_size=199.                                                                       
Training epochs on cpu:   4%|██                                                   | 39/1000 [00:10<04:57,  3.23epoch/s, loss=0.00259, prev_loss=0.00259]INFO:pykeen.stoppers.early_stopping:Stopping early after 8 evaluations at epoch 40. The best result hits_at_k=0.9874371859296482 occurred at epoch 25.  
Training epochs on cpu:   4%|██                                                   | 39/1000 [00:10<04:16,  3.75epoch/s, loss=0.00259, prev_loss=0.00259]
```

I guess the error is caused by my own data. So I checked my data and dataset 'nations'.
**1. training mapped triples in my dataset**
```
In [4]: train.mapped_triples
Out[4]: 
tensor([[    0,     1,  5404],
        [    1,     1,   320],
        [    1,     1,   340],
        ...,
        [25515,     1,  6874],
        [25515,     1,  9430],
        [25515,     1, 10406]])
```

**2. training mapped triples in dataset 'nations'**
```
In [10]: dataset.training.mapped_triples
Out[10]: 
tensor([[  0,   6,   2],
        [  0,   6,   3],
        [  0,   6,  10],
        ...,
        [ 13, 108,  12],
        [ 13, 109,   9],
        [ 13, 109,  11]])
```
It seems that the number of relation in my dataset is single and it caused the error. But I don't know the reason. Maybe I miss something about it.