Repository: pytorch
Issue: Simplification of pruned models
Labels: feature,module: nn,triaged

## 🚀 Feature

Extension of pytorch pruning utilities to effectively allow for the simplification of pruned models (i.e. actually removing zeroed out tensors from the model).

## Motivation

Pruning allows for impressive theoretical reduction of models sizes and complexity. However it usually offers little practical benefits as it is most often limited to just zeroing out weights, without actually removing the pruned filters/weights. This precludes from actual advantages provided by sparsification methods.

There seems to be a quite strong interest on the matter ([\[1\]](https://github.com/pytorch/pytorch/issues/47915) [\[2\]](https://github.com/pytorch/pytorch/issues/43552) [\[3\]](https://github.com/pytorch/pytorch/issues/36214) [\[4\]](https://github.com/pytorch/pytorch/issues/32928) [\[5\]](https://discuss.pytorch.org/t/pruning-doesnt-affect-speed-nor-memory-for-resnet-101/75814) [\[6\]](https://discuss.pytorch.org/t/discussion-of-practical-speedup-for-pruning/75212) [\[7\]](https://discuss.pytorch.org/t/save-pruned-model-after-pruning/103212) [\[8\]](https://stackoverflow.com/questions/62326683/prunning-model-doesnt-improve-inference-speed-or-reduce-model-size) and so on.)

## Pitch

We (me and **[@AndreaBrg](https://github.com/AndreaBrg)**) have currently developed a set of utilities for simplifying pruned models, and they could be ported to the pytorch core library with minor modifications. We opened this issue to gather feedback about your potential interest.

## Additional context

At the time being our codebase works with most of the torchvision models, and we observe noticeable improvements in the inference times (10 runs on cpu, with a batch size of 100, and a structured pruning ratio of 50%): results are reported in the table attached below. *Pruned time* refers to the inference time of the zeroed-out model (as obtained by prune.random_structured and prune.remove), *Simplified time* is the inference time of the corresponding simplified model. Of course, given that we actually remove entire portions of the weight matrices, we also obtain a gain in memory and storage occupation.

If you are interested in our proposal, we can further explain the technical details of our implementation.

| Architecture   | Pruned time    | Simplified time (p=50%)   |
|----------------|----------------|---------------------------|
| alexnet        | 0.2414s ± 0.0065 | 0.1064s ± 0.0010            |
| resnet18       | 1.0082s ± 0.0108 | 0.5528s ± 0.0080            |
| resnet34       | 1.6838s ± 0.0037 | 0.8636s ± 0.0022            |
| resnet50       | 3.6443s ± 0.0029 | 2.1050s ± 0.0034            |
| resnet101      | 5.5948s ± 0.0166 | 3.1392s ± 0.0033            |
| resnet152      | 7.8656s ± 0.0405 | 4.3677s ± 0.0059            |
| vgg16          | 4.8807s ± 0.0210 | 1.9711s ± 0.0272            |
| vgg16_bn       | 6.4215s ± 0.0063 | 1.9731s ± 0.0068            |
| vgg19          | 5.9035s ± 0.0445 | 2.2855s ± 0.0079            |
| vgg19_bn       | 7.5785s ± 0.0372 | 2.2698s ± 0.0106            |



cc @albanD @mruberry @jbschlosser