Repository: pytorch
Issue: `gradgradcheck` does not work with sparse inputs.
Labels: module: sparse,module: autograd,triaged

### 🐛 Describe the bug

As per title. None of the combinations below worked, and the error messages are not always relevant/helpful:

Case 1: call as is, but make sure that `fn` calls to `to_dense`:
```python
In [1]: import torch

In [2]: x = torch.rand(3, 3, dtype=torch.double).to_sparse().requires_grad_(True)

In [3]: torch.autograd.gradgradcheck(lambda x: x.to_dense(), (x,))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [3], in <cell line: 1>()
----> 1 torch.autograd.gradgradcheck(lambda x: x.to_dense(), (x,))

File ~/git/Quansight/pytorch/torch/autograd/gradcheck.py:1694, in gradgradcheck(func, inputs, grad_outputs, eps, atol, rtol, gen_non_contig_grad_outputs, raise_exception, nondet_tol, check_undefined_grad, check_grad_dtypes, check_batched_grad, check_fwd_over_rev, check_rev_over_rev, fast_mode, masked)
   1691     grad_inputs = tuple(g for g in grad_inputs if g is not None)
   1692     return grad_inputs
-> 1694 return gradcheck(
   1695     new_func, tupled_inputs + tupled_grad_outputs, eps=eps, atol=atol, rtol=rtol, raise_exception=raise_exception,
   1696     nondet_tol=nondet_tol, check_undefined_grad=check_undefined_grad,
   1697     check_grad_dtypes=check_grad_dtypes, check_batched_grad=check_batched_grad, fast_mode=fast_mode,
   1698     check_forward_ad=check_fwd_over_rev, check_backward_ad=check_rev_over_rev, masked=masked)

File ~/git/Quansight/pytorch/torch/autograd/gradcheck.py:1536, in gradcheck(func, inputs, eps, atol, rtol, raise_exception, check_sparse_nnz, nondet_tol, check_undefined_grad, check_grad_dtypes, check_batched_grad, check_batched_forward_grad, check_forward_ad, check_backward_ad, fast_mode, masked)
   1534         return False
   1535 else:
-> 1536     return _gradcheck_helper(**args)

File ~/git/Quansight/pytorch/torch/autograd/gradcheck.py:1547, in _gradcheck_helper(func, inputs, eps, atol, rtol, check_sparse_nnz, nondet_tol, check_undefined_grad, check_grad_dtypes, check_batched_grad, check_batched_forward_grad, check_forward_ad, check_backward_ad, fast_mode, masked)
   1545 func_out = func(*tupled_inputs)
   1546 outputs = _differentiable_outputs(func_out)
-> 1547 _check_outputs(outputs)
   1549 gradcheck_fn = functools.partial(_fast_gradcheck if fast_mode else _slow_gradcheck, masked=masked)
   1550 _gradcheck_real_imag(gradcheck_fn, func, func_out, tupled_inputs, outputs, eps,
   1551                      rtol, atol, check_grad_dtypes, check_forward_ad=check_forward_ad,
   1552                      check_backward_ad=check_backward_ad, nondet_tol=nondet_tol,
   1553                      check_undefined_grad=check_undefined_grad)

File ~/git/Quansight/pytorch/torch/autograd/gradcheck.py:772, in _check_outputs(outputs)
    768 def _check_outputs(outputs) -> None:
    769     if any(_is_sparse_any_tensor(t) for t in outputs if isinstance(t, torch.Tensor)):
    770         # it is easier to call to_dense() on the sparse output than
    771         # to modify analytical jacobian
--> 772         raise ValueError('Sparse output is not supported at gradcheck yet. '
    773                          'Please call to_dense() on the output of fn for gradcheck.')
    774     if any(t.layout == torch._mkldnn for t in outputs if isinstance(t, torch.Tensor)):  # type: ignore[attr-defined]
    775         raise ValueError('MKLDNN output is not supported at gradcheck yet. '
    776                          'Please call to_dense() on the output of fn for gradcheck.')

ValueError: Sparse output is not supported at gradcheck yet. Please call to_dense() on the output of fn for gradcheck.

```

Case 2: specify `masked=True`.
```python
In [4]: torch.autograd.gradgradcheck(lambda x: x.to_dense(), (x,), masked=True)
---------------------------------------------------------------------------
GradcheckError                            Traceback (most recent call last)
Input In [4], in <cell line: 1>()
----> 1 torch.autograd.gradgradcheck(lambda x: x.to_dense(), (x,), masked=True)

File ~/git/Quansight/pytorch/torch/autograd/gradcheck.py:1694, in gradgradcheck(func, inputs, grad_outputs, eps, atol, rtol, gen_non_contig_grad_outputs, raise_exception, nondet_tol, check_undefined_grad, check_grad_dtypes, check_batched_grad, check_fwd_over_rev, check_rev_over_rev, fast_mode, masked)
   1691     grad_inputs = tuple(g for g in grad_inputs if g is not None)
   1692     return grad_inputs
-> 1694 return gradcheck(
   1695     new_func, tupled_inputs + tupled_grad_outputs, eps=eps, atol=atol, rtol=rtol, raise_exception=raise_exception,
   1696     nondet_tol=nondet_tol, check_undefined_grad=check_undefined_grad,
   1697     check_grad_dtypes=check_grad_dtypes, check_batched_grad=check_batched_grad, fast_mode=fast_mode,
   1698     check_forward_ad=check_fwd_over_rev, check_backward_ad=check_rev_over_rev, masked=masked)

File ~/git/Quansight/pytorch/torch/autograd/gradcheck.py:1536, in gradcheck(func, inputs, eps, atol, rtol, raise_exception, check_sparse_nnz, nondet_tol, check_undefined_grad, check_grad_dtypes, check_batched_grad, check_batched_forward_grad, check_forward_ad, check_backward_ad, fast_mode, masked)
   1534         return False
   1535 else:
-> 1536     return _gradcheck_helper(**args)

File ~/git/Quansight/pytorch/torch/autograd/gradcheck.py:1543, in _gradcheck_helper(func, inputs, eps, atol, rtol, check_sparse_nnz, nondet_tol, check_undefined_grad, check_grad_dtypes, check_batched_grad, check_batched_forward_grad, check_forward_ad, check_backward_ad, fast_mode, masked)
   1539 def _gradcheck_helper(func, inputs, eps, atol, rtol, check_sparse_nnz, nondet_tol, check_undefined_grad,
   1540                       check_grad_dtypes, check_batched_grad, check_batched_forward_grad, check_forward_ad,
   1541                       check_backward_ad, fast_mode, masked):
   1542     tupled_inputs = _as_tuple(inputs)
-> 1543     _check_inputs(tupled_inputs, check_sparse_nnz, masked)
   1545     func_out = func(*tupled_inputs)
   1546     outputs = _differentiable_outputs(func_out)

File ~/git/Quansight/pytorch/torch/autograd/gradcheck.py:732, in _check_inputs(tupled_inputs, check_sparse_nnz, masked)
    730 def _check_inputs(tupled_inputs, check_sparse_nnz, masked) -> bool:
    731     if masked and not check_sparse_nnz and any(_is_sparse_any_tensor(t) for t in tupled_inputs if isinstance(t, torch.Tensor)):
--> 732         raise GradcheckError('gradcheck expects all tensor inputs are dense'
    733                              ' when check_sparse_nnz is set to False and masked is set to True.')
    734     # Make sure that gradients are saved for at least one input
    735     any_input_requiring_grad = False

GradcheckError: gradcheck expects all tensor inputs are dense when check_sparse_nnz is set to False and masked is set to True.
```
Case 3: well, if `masked=True` does not work, we should use both `masked=True` and `check_sparse_nnz=True`, but:
```python
In [5]: torch.autograd.gradgradcheck(lambda x: x.to_dense(), (x,), masked=True, check_sparse_nnz=True)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Input In [5], in <cell line: 1>()
----> 1 torch.autograd.gradgradcheck(lambda x: x.to_dense(), (x,), masked=True, check_sparse_nnz=True)

TypeError: gradgradcheck() got an unexpected keyword argument 'check_sparse_nnz'

```

### Versions

Current master.

cc @alexsamardzic @pearu @cpuhrsch @amjames @bhosmer @ezyang @albanD @zou3519 @gqchen @soulitzer @Lezcano @Varal7