Repository: pytorch
Issue: `torch.sparse_csc_tensor` matrix multiplication produces MKL error SPARSE_STATUS_ALLOC_FAILED when density is too high
Labels: module: sparse,triaged

### 🐛 Describe the bug

I create 2 random sparse matrices with `s` elements in each column, and perform matrix multiplication. I do this in AWS SageMaker Studio jupyter notebook, on a large instance (ml.r5.8xlarge with 32 vCPUs and 256GB RAM). 

My code:
```
import numpy as np
import pandas as pd
import sys, torch
print(f"Python {sys.version.split()[0]}   torch {torch.__version__}")

def rand_mat_sp(m, n, s):
    """Create a sparse mxn matrix, in which every column has exactly s nonzero entries, with randomly chosen positions and random values from the interval (0,1]."""
    I = np.concatenate([[0],np.full(n,s)]).cumsum().astype(np.int32)
    uu = np.zeros(n*s, dtype=np.int32)
    for j in range(n):  uu[j*s:(j+1)*s] = np.sort(np.random.default_rng().choice(m, size=s, replace=False, shuffle=False))
    ww = (1-np.random.rand(n*s)).astype(np.float32)  # entries have uniform distribution on interval (0,1]
    return torch.sparse_csc_tensor(I,uu,ww,size=(m,n),dtype=torch.float32)
print(rand_mat_sp(5,5,3).to_dense())

for s in [150,160]:
    %time x, y = [rand_mat_sp(10**5,10**5,s) for i in [1,2]]
    %time z = torch.mm(x,y)  # matrix multiplication
    print(z._nnz()/10**5)  # nr nonzero entries per column
```

When `s` is too large, the code produces the error:
```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<timed exec> in <module>

RuntimeError: MKL error: SPARSE_STATUS_ALLOC_FAILED when calling `mkl_sparse_spmm( SPARSE_OPERATION_NON_TRANSPOSE, mkl_sparse_mat1.descriptor(), mkl_sparse_mat2.descriptor(), &result_desc)`
20161.82454
```
Note that doing the same multiplication with `scipy.sparse.csc_array` instead, produces no error, for arbitrarily large `s`. 
Screenshot:
![Screenshot at 2023-04-27 11-05-30](https://user-images.githubusercontent.com/42877636/234818217-98405904-846a-4003-be30-f717ee6e20ec.png)


### Versions

Code:
```
! wget https://raw.githubusercontent.com/pytorch/pytorch/main/torch/utils/collect_env.py
%run collect_env.py
```
Output:
```
--2023-04-27 09:22:48--  https://raw.githubusercontent.com/pytorch/pytorch/main/torch/utils/collect_env.py
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.109.133, 185.199.108.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 21550 (21K) [text/plain]
Saving to: ‘collect_env.py.2’

collect_env.py.2    100%[===================>]  21.04K  --.-KB/s    in 0s      

2023-04-27 09:22:48 (124 MB/s) - ‘collect_env.py.2’ saved [21550/21550]

Collecting environment information...
PyTorch version: 2.0.0+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.6 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.26.3
Libc version: glibc-2.31

Python version: 3.9.16 | packaged by conda-forge | (main, Feb  1 2023, 21:39:03)  [GCC 11.3.0] (64-bit runtime)
Python platform: Linux-4.14.311-233.529.amzn2.x86_64-x86_64-with-glibc2.31
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   46 bits physical, 48 bits virtual
CPU(s):                          32
On-line CPU(s) list:             0-31
Thread(s) per core:              2
Core(s) per socket:              16
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           85
Model name:                      Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
Stepping:                        7
CPU MHz:                         3072.166
BogoMIPS:                        4999.99
Hypervisor vendor:               KVM
Virtualization type:             full
L1d cache:                       512 KiB
L1i cache:                       512 KiB
L2 cache:                        16 MiB
L3 cache:                        35.8 MiB
NUMA node0 CPU(s):               0-31
Vulnerability Itlb multihit:     KVM: Vulnerable
Vulnerability L1tf:              Mitigation; PTE Inversion
Vulnerability Mds:               Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Meltdown:          Mitigation; PTI
Vulnerability Mmio stale data:   Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Retbleed:          Vulnerable
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Retpolines, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] sagemaker-pytorch-training==2.7.0
[pip3] torch==2.0.0
[pip3] torchaudio==0.13.1+cpu
[pip3] torchdata==0.5.1
[pip3] torchvision==0.14.1+cpu
[pip3] triton==2.0.0
[conda] mkl                       2022.2.1         h84fe81f_16997    conda-forge
[conda] mkl-include               2023.0.0         h84fe81f_26648    conda-forge
[conda] numpy                     1.23.5                   pypi_0    pypi
[conda] sagemaker-pytorch-training 2.7.0                    pypi_0    pypi
[conda] torch                     2.0.0                    pypi_0    pypi
[conda] torchaudio                0.13.1+cpu               pypi_0    pypi
[conda] torchdata                 0.5.1                    pypi_0    pypi
[conda] torchvision               0.14.1+cpu               pypi_0    pypi
[conda] triton                    2.0.0                    pypi_0    pypi
```

cc @alexsamardzic @nikitaved @pearu @cpuhrsch @amjames @bhosmer