Repository: pykeen/pykeen
Issue: Trying to load a model from a pipeline checkpoint - dimension issues
Root cause: Documentation Deficiency

I am trying to load a model from a pipeline checkpoint object for testing inference whilst the pipeline is still running. However, when I try to load the model_state_dict, I get a dimension mismatch error relating to the number of entities. Is the reloading of models from the pipeline checkpoints supported? (I note it isn't demonstrated in the documentation but clearly must be happening if you are resuming training in the pipeline object -- although I had a very quick look in the code and couldn't see where the model weights are being reloaded).

Anyway I have run a pipeline as follows (I am using my own dataset and I wonder if this is part of the issue):

``` python import pandas as pd
from pykeen.pipeline import pipeline
from pykeen.triples import TriplesFactory

df_train = pd.read_csv("edges.csv")
df_val = df_train[df_train['r'] == 'blah'].sample(frac=0.2)
df_train = df_train.drop(df_val.index)
df_test = pd.read_csv("test.csv")

training = TriplesFactory.from_labeled_triples(df_train[['h', 'r', 't']].values)
val = TriplesFactory.from_labeled_triples(df_val[['h', 'r', 't']].values, entity_to_id=training.entity_to_id,relation_to_id=training.relation_to_id)
test = TriplesFactory.from_labeled_triples(df_test[['h', 'r', 't']].values, entity_to_id=training.entity_to_id,relation_to_id=training.relation_to_id)

result = pipeline(
    training=training,
    testing=test,
    validation=val,
    model='RotatE',
    training_kwargs=dict(
        checkpoint_name='rotate_checkpoint.pt',
    ),) 
```

This model has been training for sometime so I thought that I would load the checkpoint to see how it was getting on. This was done as follows:

``` python import torch
from pykeen.models import RotatE

checkpoint = torch.load("rotate_checkpoint.pt")
pykeen_model = RotatE(triples_factory=training)
pykeen_model.load_state_dict(checkpoint['model_state_dict'])
``` 

However at this stage I get an error complaining that the entity embedding matrices do not match in size - it's off by 4 if that is relevant at all, with the new model being the smaller of the two. Any thoughts on if I am doing something incorrectly? 


So digging into this a little bit further, I am getting confused over the expected dimensionality of the entity embedding matrix. I looked at the value of `training.num_entities` which gave a value of say 100,00. However the saved embedding matrix in the pipeline checkpoint has a size of 100,011 , whilst the size of the matrix in the `pykeen_model` created above is 100,015 .

Hi @sbonner0 
Looking at your code my first suspicion would be that you use a random sampler to create a validation dataset while you in the subsequent step drop these triples from the training dataset, which effectively makes your training dataset a random 80% pick from your full dataset. As the `TriplesFactory` derives the required number of entities from the number of distinct entities in the training dataset, the variations you describe in https://github.com/pykeen/pykeen/issues/482#issuecomment-857704112 are likely to be caused by the varying number of distinct entities in your sampled training dataset.

A way to mitigate this problem could be to fix the seed of the sampler by changing your code to
```df_val = df_train[df_train['r'] == 'blah'].sample(frac=0.2, random_state=10)```

Does this solve the issue?

Hey thanks @lvermue that solved the issue thank you! Sorry should have been able to catch that on my own. 

As an aside, It could be cool to add this functionality to the documentation? I'm sure users running multi-day jobs would find it interesting to load a partially trained model for some experimentation. Happy to make a PR with this added if you think it would be useful. 

I think https://pykeen.readthedocs.io/en/latest/tutorial/checkpoints.html has you covered

Thanks @cthoyt this is a great resource. However I don't think it explicitly covers the reloading of a model checkpoint for inference use which is covered by this code block:

``` python
from pykeen.models import RotatE

checkpoint = torch.load("rotate_checkpoint.pt")
pykeen_model = RotatE(triples_factory=training)
pykeen_model.load_state_dict(checkpoint['model_state_dict'])
```

Note this is different from the normal model reloading process. 

Hi @sbonner0 
I have had a chat with @mberr and @mali-git and we deem this functionality useful (@cthoyt I assumed your thumbs up to be a yes vote).
I have proposed a PR #498 based on this issue, which is accompanied with a tutorial on how to use it.
If you see anything missing or have additional ideas, please feel free to add your comments to the linked PR.

Hi @lvermue - I've been through PR #498 and it looks great. Automatic saving of the entity and relation types for custom datasets is going to improve the use of checkpoints a lot - thanks!

Just finished up editing/revising on the tutorial. Is the tutorial written by @lvermue sufficient to close this issue? If so, we will change the text in PR to reflect that, then it's ready to merge!

Hi @cthoyt, yes absolutely the new tutorial is more than enough to close this!