Repository: pytorch
Issue: SparseAdam: working with dense parameters but sparse gradients - usecase 
Labels: module: optimizer,triaged,actionable

### 📚 The doc issue

I have a use case where some gradients are sparse while others are dense, and therefore I'm splitting them into two groups and optimizing them separately using Adam and AdamSparse. 

In the case of the parameter tensor that is dense while the gradient tensor is sparse (many of the values are set to zero as I do not want to update those parameters), which seems like a perfect use case for the SparseAdam optimizer, however, I get into the situation where if I pass my parameters associated with the sparse gradient matrix (when I say sparse here, I mean that many of the values are zero, not that the gradient tensor is converted to a sparse tensor using `torch.sparse_coo_tensor` or `torch.to_sparse`. 

According to the documentation of the SparseAdam optimizer the variant is applied for the following: 

> In this variant, only moments that appear in the gradient get updated, and only those portions of the gradient get applied to the parameters.

And according to the source code, in the `init()` method, sparse parameters are rejected after being checked with the `param.is_sparse`:
`if sparse_params:
      raise ValueError(
                f"Sparse params at indices {sparse_params}: SparseAdam requires dense parameter tensors"
            )
`
So the optimizer only accepts dense parameters. However, it does not accept dense gradients (which is the purpose of the optimizer). But the problem I am having here, is if I am passing my dense parameters with the sparse gradient tensor (many zeros) the optimizer fails at `optimizer.step` because of this check 

` if not p.grad.is_sparse:
       raise RuntimeError('SparseAdam does not support dense gradients, please consider Adam instead')`

A possible work around is using a hook to change the parameter gradient to a sparseTensor using `torch.sparse_coo_tensor`  but then i get the error 

`Exception has occurred: RuntimeError
hook 'hook' has changed the type of value (was torch.cuda.FloatTensor got torch.cuda.sparse.FloatTensor)`

It is unclear from the documentation how this optimizer is supposed to be used or if this use case can be covered by the sparseAdam optimizer. 

What I would think a possibility of this use-case being covered, which might be useful in many cases where parts of the layers/nodes would be updated, but not others, is passing the parameters and their zero gradients to the sparseAdam optimizer. The optimizer converts the gradient to a sparse tensor to enable this use case. 

If there's a workaround, I'd appreciate the help/input. 



### Suggest a potential alternative/fix

How the sparseAdam optimizer is used - which use cases does it cover, and which use cases it does not. The documentation mentions that it's used for sparse tensors, but not much else. It is specifically confusing because the source code has a check for if the parameter dense is dense (which is required) and the gradient tensor is sparse (which is required), but not on how to satisfy this use-case. 

cc @vincentqb @jbschlosser @albanD @janeyx99