Repository: pykeen/pykeen
Issue: How to summarize number of parameters in a model?
GitHub labels: question

Hi, I would like to know if there is any easy way i can obtain a summary of a pykeen model, similar to what keras summary function does (list number of  parameters, number of trainable parameters, non-trainable parameters, etc).


Since i couldn't seem to find anything in the docs, I tried to use 2 external packages (listed bellow) in order to obtain such functionality but nothing seems to work.

### [torchsummary](https://github.com/sksq96/pytorch-summary)

This package has issues dealing with  nn.Embedding  https://github.com/sksq96/pytorch-summary/issues/42, and is at the moment still an open issue.

### [torchsummaryX ](https://github.com/nmhkahn/torchsummaryX) 

I also tried torchsummaryX  wich was proposed in the issue thread of the library above as an alternative, however its seems like it internally calls model.forward() wich with pykeen's  Models raises an  NotImplementedError

``` python
from pykeen.datasets import Nations
dataset = Nations()
training_triples_factory = dataset.training

from pykeen.models import TransE
transe_mod = TransE(triples_factory=training_triples_factory, embedding_dim=10)

transe_mod.forward()
```
Gives error:
> ---------------------------------------------------------------------------
> NotImplementedError                       Traceback (most recent call last)
> <ipython-input-17-a16fd01968b2> in <module>()
> ----> 1 transe_mod.forward()
> 
> /usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _forward_unimplemented(self, *input)
>     199         registered hooks while the latter silently ignores them.
>     200     """
> --> 201     raise NotImplementedError
>     202 
>     203 
> 
> NotImplementedError: 



------

I would like to ask you, if you know of any way I can do something like this. It is of course not a crucial feature (since i can calculate "by hand" this statistics), but it is something nice to have especially when implementing a new model (in order to easily compare its complexity with others).

Best regards,

Rodrigo Pereira

The only thing we've implemented ourselves for this (which we present in the final figures of our [benchmarking paper](https://arxiv.org/abs/2006.13365)) is for counting the number of bytes that a model takes up. This was useful for showing that there are at best diminishing returns for using models like ConvKB and ConvE since there are other models that are simpler yet as performant. That code is here:

https://github.com/pykeen/pykeen/blob/5a24de7d6c42af36ea1a546c10e68634da817852/src/pykeen/models/base.py#L242-L248

I'm not really sure what's going on with your example, but the `forward()` function needs to take in a 2-tensor of longs that correspond to a list of triples. I don't see where `torchsummaryX` is used in your example or how calling the forward function is part of it (I don't have time to look into `torchsummaryX` at the moment)

If you have some ideas on how to generally calculate these statistics given a model (or maybe just start simpler, with one of the loss functions or other components), we would love to have a PR for it.


Do parameters have a flag for being trainable? Is that the same as the gradient being enabled? In that case you could modify this code for the bytes to do some filtering and maybe get what you want

Hi @Rodrigo-A-Pereira ,

you get an overview of the model structure by simply printing the object, e.g.,
```python-console
>>> from pykeen.datasets import get_dataset
>>> dataset = get_dataset(dataset="nations")
>>> from pykeen.models import ConvE
>>> model = ConvE(triples_factory=dataset.training)
>>> print(model)
ConvE(
  (loss): BCEAfterSigmoidLoss()
  (regularizer): NoRegularizer()
  (entity_embeddings): Embedding(
    (_embeddings): Embedding(14, 200)
  )
  (relation_embeddings): Embedding(
    (_embeddings): Embedding(55, 200)
  )
  (bias_term): Embedding(
    (_embeddings): Embedding(14, 1)
  )
  (inp_drop): Dropout(p=0.2, inplace=False)
  (hidden_drop): Dropout(p=0.3, inplace=False)
  (feature_map_drop): Dropout2d(p=0.2, inplace=False)
  (conv1): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1))
  (bn0): BatchNorm2d(1, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (bn2): BatchNorm1d(200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (fc): Linear(in_features=10368, out_features=200, bias=True)
)
```

To get information about the model's parameters, you can use Pytorch's builtin [`named_parameters`](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.named_parameters)
```python-console
>>> for name, param in model.named_parameters():
>>>     print(name, tuple(param.shape), param.numel(), param.requires_grad, sep="\t")
entity_embeddings._embeddings.weight	(14, 200)	2800	True
relation_embeddings._embeddings.weight	(55, 200)	11000	True
bias_term._embeddings.weight	(14, 1)	14	True
conv1.weight	(32, 1, 3, 3)	288	True
conv1.bias	(32,)	32	True
bn0.weight	(1,)	1	True
bn0.bias	(1,)	1	True
bn1.weight	(32,)	32	True
bn1.bias	(32,)	32	True
bn2.weight	(200,)	200	True
bn2.bias	(200,)	200	True
fc.weight	(200, 10368)	2073600	True
fc.bias	(200,)	200	True
```

Besides parameters, PyTorch modules can also have buffers, which are essentially what keras calls non-trainable parameters such as, e.g., the batch normalization running averages. You can access them via Pytorch's [`named_buffers`](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.named_buffers)
```python-console
>>> for name, buf in model.named_buffers():
>>>     print(name, tuple(buf.shape), buf.numel(), sep="\t")
regularizer.weight	()	1
regularizer.regularization_term	(1,)	1
bn0.running_mean	(1,)	1
bn0.running_var	(1,)	1
bn0.num_batches_tracked	()	1
bn1.running_mean	(32,)	32
bn1.running_var	(32,)	32
bn1.num_batches_tracked	()	1
bn2.running_mean	(200,)	200
bn2.running_var	(200,)	200
bn2.num_batches_tracked	()	1
```

Depending of what you want to do with these information, you can also put them, e.g., into a pandas Dataframe instead of printing them to console.

Thank you,

I knew about the structure provided by printing the model. But I didn't know pytorch had built in functions for parameters. It is very helpfull.

Best regards, 

Rodrigo Pereira