Repository: pytorch
Issue: torch.sparse improvements - tracking issue
Labels: high priority,module: sparse,triaged

This tracking is meant to list the main improvements we'll be focusing on for sparse tensors, and then mapping them to the releases they should land in. _Please note: these are estimates and indicate current prioritization and require effort - don't rely on this, things may still shift around._

The fill-value proposal (https://github.com/pytorch/rfcs/pull/8) has a good description of the different interpretations of sparse tensors: linear algebra (where unspecified values are assumed to be 0), graph algorithms (where sparse structure indicates nodes/edges), and calculus (where non-zero fill values are needed). _The description of application-level use cases for each of those interpretations is still TODO_.

**~1.7~ 1.8 release**

Sparse documentation updates:

- [x] gh-44635: review and update all content on https://pytorch.org/docs/stable/sparse.html --> PR gh-44635
- [ ] Then update the _"This API is in beta, ..."_ warning at the top of the docs. Feature classification may still be "beta", but API should be stable. Add performance-related note instead.
- [ ] Add an [overview using tables](https://github.com/Quansight-Labs/rfcs/blob/pearu/rfc0005/RFC0003-sparse-roadmap/SparseSupportState.md#nnactivation-functions) of _all_ operators in PyTorch, and show whether they support sparse tensors on CPU, on CUDA, and have autograd support.

**1.8 release**

- [ ] CSR/GCS format (see ~https://github.com/pytorch/rfcs/pull/4, gh-44190~, https://github.com/pytorch/rfcs/pull/13, gh-50937, )
- [x] Performance benchmarks for matrix-matrix and matrix-vector ops (dense-sparse, sparse-sparse, and compare to dense-dense) (see gh-51647)
- [x] Benchmarking framework for sparse that's easy to extend when sparse support is added to new operators (see gh-48397)
- [ ] Sparse test infrastructure improvements (see [here for details](https://github.com/pytorch/pytorch/pull/43330#issuecomment-691210906)) - PR gh-49158
- [ ] Add sparse tensor support for all element-wise functions that map 0 to 0 (this is low-hanging fruit)
- [x] https://github.com/pytorch/pytorch/issues/45113
- [ ] https://github.com/pytorch/pytorch/issues/45897
- [ ] https://github.com/pytorch/pytorch/issues/45996
- [ ] https://github.com/pytorch/pytorch/issues/14945
- [ ] Create public API to replace use of `_indices`, `_values`, `_nnz`. See gh-45695.

**1.9 release**

- [ ] fill-value support in COO and GCS sparse tensors (see https://github.com/pytorch/rfcs/pull/8)
- [ ] extend fill-value support to all functions currently supporting sparse

**To (re)assess**

- [ ] Block sparse tensors, gh-9222

**Relevant but blocked**

- [ ] Sparse-Dense elementwise Multiplication, gh-3158

**CSR sparse tensors**

- [The state of CSR sparse tensor support as of March 2021](https://pearu.github.io/csr_tensor_support.html)
- [x] #50937 Sparse tensor CSR layout, CPU only
- [x] #56485 Sparse tensor CSR layout for CUDA
- [ ] CSR sparse tensor:  ops not implemented yet
   - [ ]  #59060 Addition of sparse CSR tensor
   - [ ]  csr_sparse @ csr_sparse op 
- Testing CSR support
  - [x] https://github.com/pytorch/pytorch/issues/56371 - modernize CSR test-suite
  - [x] https://github.com/pytorch/pytorch/issues/56369 - eliminate global usage of torch.set_default_dtype

- [ ] cuSPARSE backend: https://github.com/pytorch/pytorch/issues/60854
- [ ] MKL backend: https://github.com/pytorch/pytorch/issues/60858

 
cc @ezyang @gchanan @zou3519 @vincentqb @aocsa @nikitaved @pearu @v0dro @mruberry