Repository: pykeen/pykeen
Issue: Question about num_training_instances on loss calculation
Root cause: Validation Gap

Hi, I have a question on how the loss is calculated, in specific what is this `num_training_instances` variable:

https://github.com/pykeen/pykeen/blob/e33cdc7dd9c52b51f41cb347a7bb1fc4394b345b/src/pykeen/training/training_loop.py#L664-L665

To give a bit of context on the reasoning for my question, at the moment I am trying to log the 'performance' of the model on the validation set during training, in order to check for cases of overfitting. Adding an early stopper to the pipeline was giving me these logs, however the evaluation steps were very slow and were the bottleneck of the training times (sometimes one evaluation taking as much time as training for 10 epochs). As such i decided to add a way to only log the loss value of a forward pass of the validation set through the model, however I am a bit unsure on how the previous line of code is calculating the loss.

Hi @Rodrigo-A-Pereira ,

if we use `"mean"` reduction, we want to compute the average loss per sample. 

Notice that in case of incomplete last batches in an epoch, computing the mean per batch, and then the mean over batches, would give a higher weight to those samples contained in the last batch. To compensate for this, we add a somewhat hidden compension, where we multiply the mean loss per batch by the batch size
https://github.com/pykeen/pykeen/blob/e33cdc7dd9c52b51f41cb347a7bb1fc4394b345b/src/pykeen/training/training_loop.py#L788-L791
and then finally divide the epoch loss by the number of samples instead of the number of batches.
https://github.com/pykeen/pykeen/blob/e33cdc7dd9c52b51f41cb347a7bb1fc4394b345b/src/pykeen/training/training_loop.py#L664-L665

P.S.: I updated your issue description to use a perma-link to the referenced code line (cf. https://docs.github.com/en/github/writing-on-github/working-with-advanced-formatting/creating-a-permanent-link-to-a-code-snippet). They are better than just linking to line XYZ on a branch, since these lines may change in the future. As a nice side effect, GitHub will render those lines automatically for you :slightly_smiling_face: 

@mberr , thank you for the explanation!

I have one other question in the sequence of the use case i described. Is there any "easier" way to make a forward pass with the validation set "through the training_loop" and obtain the loss, while not "train" the model (in mode model.eval()), than to implement an alternative to the `_train()` function?


Yes, by using a [callback](https://github.com/pykeen/pykeen/blob/e33cdc7dd9c52b51f41cb347a7bb1fc4394b345b/src/pykeen/training/callbacks.py#L70-L71).

The following may serve as a starting point:

```python
from typing import Any
import torch
import torch.utils.data
from pykeen.training import TrainingLoop, TrainingCallback
from pykeen.triples.instances import SLCWAInstances


class ValidationCallback(TrainingCallback):
    def __init__(self, validation_instances: SLCWAInstances, batch_size: int):
        super().__init__()
        self.validation_instances = validation_instances
        self.batch_size = batch_size

    @torch.inference_mode()
    def post_epoch(self, epoch: int, epoch_loss: float, **kwargs: Any) -> None:  # noqa: D102
        self.training_loop: TrainingLoop
        self.model.eval()
        loss = (
            sum(
                self.training_loop._process_batch(
                    batch=batch,
                    start=None,
                    stop=None,
                )
                for batch in torch.utils.data.DataLoader(
                    dataset=self.validation_instances,
                    batch_size=self.batch_size,
                    pin_memory=True,
                )
            )
            / len(self.validation_instances)
        )
        self.model.train()

        print(loss)  # or do something else
```

Thank you very much @mberr! I'll try that.

Hi @mberr, sorry to bother again, but a question surged when i was checking how the callbacks were implemented.

Not sure if this intended, but shouldn't the `kwargs` be transmitted to the multiple registered callbacks on the `MultiTrainingCallback` class, otherwise the registered callbacks can't access any extra arguments.

https://github.com/pykeen/pykeen/blob/e33cdc7dd9c52b51f41cb347a7bb1fc4394b345b/src/pykeen/training/callbacks.py#L214-L232

Something like this:
```python 
def post_epoch(self, epoch: int, epoch_loss: float, **kwargs: Any) -> None:  # noqa: D102 
     for callback in self.callbacks: 
         callback.post_epoch(epoch=epoch, epoch_loss=epoch_loss, kwargs) 
```

That's a good catch, @Rodrigo-A-Pereira ! Callbacks are quite new in pykeen (#335 ) and thus not as thoroughly tested as other components yet :sweat_smile: 

It would be great, if you'd open a PR with your fix for it (except it should be `**kwargs` if I am not mistaken, cf. `pre_step` where it seems to be correct).

Thanks for taking the time to answering my questions! Keep up the good work.