Repository: pytorch
Issue: Slower performance of `torch.mm` method with sparse CSR tensor
Labels: module: sparse,triaged,module: mkl

### 🐛 Describe the bug

In pytorch version 1.11.0 `torch.mm` method performs slower for sparse CSR matrices than in pytorch version 1.9.0.
This issue can be reproduced by using the following code example on both pytorch versions:

```
import torch
import timeit

print(torch.__version__)

torch.manual_seed(1233)
crows = torch.randint(0, 50, (1000000,)).cumsum(0).int()
crows[0] = 0
cols = torch.randint(0, 100000, (crows[-1],)).sort()[0].int()
mock_vals = torch.rand_like(cols, dtype=torch.float)

if hasattr(torch, "sparse_csr_tensor"):
    sparse_csr_tensor = torch.sparse_csr_tensor
else:
    sparse_csr_tensor = torch._sparse_csr_tensor

csr_mat = sparse_csr_tensor(crow_indices = crows, col_indices=cols, values=mock_vals)
dense_mat = torch.rand(csr_mat.shape[-1], 20)

print(timeit.timeit("torch.mm(csr_mat, dense_mat)", globals=globals(), number=100))
```

This test has been performed on several devices. It takes at best approximately 8% longer to perform matrix multiplication in pytorch version `1.11.0+cpu` than in `1.9.0+cpu`. This results in significant speed downgrade for applications that rely on sparse matrix multiplication.

### Versions

```
Collecting environment information...
PyTorch version: 1.9.0+cpu
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.3 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.8.10 (default, Nov 26 2021, 20:14:08)  [GCC 9.3.0] (64-bit runtime)
Python platform: Linux-5.4.0-94-generic-x86_64-with-glibc2.29
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy==0.770
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.18.5
[pip3] torch==1.9.0+cpu
[pip3] torch-scatter==2.0.9
[pip3] torch-sparse==0.6.12
[pip3] torch-tb-profiler==0.3.1
[pip3] torchaudio==0.9.0
[pip3] torchvision==0.10.0+cpu
[conda] Could not collect
```

```
Collecting environment information...
PyTorch version: 1.11.0+cpu
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.3 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.8.10 (default, Nov 26 2021, 20:14:08)  [GCC 9.3.0] (64-bit runtime)
Python platform: Linux-5.4.0-94-generic-x86_64-with-glibc2.29
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy==0.770
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.18.5
[pip3] torch==1.11.0+cpu
[pip3] torch-scatter==2.0.9
[pip3] torch-sparse==0.6.12
[pip3] torch-tb-profiler==0.3.1
[pip3] torchaudio==0.9.0
[pip3] torchvision==0.10.0+cpu
[conda] Could not collect
```

cc @nikitaved @pearu @cpuhrsch