Repository: pytorch
Issue: Cannot call CUDAGeneratorImpl::current_seed during CUDA graph capture
Labels: module: cuda,triaged

### 🐛 Describe the bug

When attempting to use
```
model = torch.cuda.make_graphed_callables(model, (rand_data,))
```
and our model contains checkpoints or sequential checkpoints like this:
```
x = checkpoint(self.layer1, x, use_reentrant=False)
```
I got this error
```
Traceback (most recent call last):
  File "test_resnet.py", line 11, in <module>
    model = torch.cuda.make_graphed_callables(model, (rand_data,))
  File "/mnt/cache/share/spring/conda_envs/miniconda3/envs/s0.3.5/lib/python3.7/site-packages/torch/cuda/graphs.py", line 279, in make_graphed_callables
    outputs = func(*args)
  File "/mnt/cache/share/spring/conda_envs/miniconda3/envs/s0.3.5/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/mnt/lustre/chendingyu1/resnet.py", line 285, in forward
    return self._forward_impl(x)
  File "/mnt/lustre/chendingyu1/resnet.py", line 273, in _forward_impl
    x = checkpoint(self.layer1, x, use_reentrant=False)
  File "/mnt/cache/share/spring/conda_envs/miniconda3/envs/s0.3.5/lib/python3.7/site-packages/torch/utils/checkpoint.py", line 240, in checkpoint
    *args
  File "/mnt/cache/share/spring/conda_envs/miniconda3/envs/s0.3.5/lib/python3.7/site-packages/torch/utils/checkpoint.py", line 333, in _checkpoint_without_reentrant
    fwd_gpu_devices, fwd_gpu_states = get_device_states(*args)
  File "/mnt/cache/share/spring/conda_envs/miniconda3/envs/s0.3.5/lib/python3.7/site-packages/torch/utils/checkpoint.py", line 44, in get_device_states
    fwd_gpu_states.append(torch.cuda.get_rng_state())
  File "/mnt/cache/share/spring/conda_envs/miniconda3/envs/s0.3.5/lib/python3.7/site-packages/torch/cuda/random.py", line 31, in get_rng_state
    return default_generator.get_state()
RuntimeError: Cannot call CUDAGeneratorImpl::current_seed during CUDA graph capture. If you need this call to be captured, please file an issue. Current cudaStreamCaptureStatus: cudaStreamCaptureStatusActive
```
Setting `preserve_rng_state=False` seems to get around this problem, but it will behave differently when recomputing activations.
If I use `use_reentrant=True` in checkpoint function, I got the following error:

```RuntimeError: Checkpointing is not compatible with .grad() or when an `inputs` parameter is passed to .backward(). Please use .backward() and do not pass its `inputs` argument.```

Can I use checkpoint in cuda graph without setting `preserve_rng_state=False` or `use_reentrant=False`? As the trackback says, "If you need this call to be captured, please file an issue".

### Versions

PyTorch version: 1.11.0+cu113
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: CentOS Linux 7 (Core) (x86_64)
GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)
Clang version: Could not collect
CMake version: version 2.8.12.2
Libc version: glibc-2.17

Python version: 3.7.11 (default, Jul 27 2021, 14:32:16)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-3.10.0-957.el7.x86_64-x86_64-with-centos-7.6.1810-Core
Is CUDA available: True
CUDA runtime version: 9.0.176

Nvidia driver version: 460.32.03

Versions of relevant libraries:
[pip3] numpy==1.21.5
[pip3] spring==0.7.2+cu112.torch1110.mvapich2.nartgpu.develop.805601a8
[pip3] torch==1.11.0+cu113
[pip3] torchvision==0.12.0+cu113
[conda] numpy                     1.21.5                   pypi_0    pypi
[conda] spring                    0.7.0+cu112.torch1110.mvapich2.pmi2.nartgpu          pypi_0    pypi
[conda] torch                     1.11.0+cu113             pypi_0    pypi
[conda] torchvision               0.12.0+cu113             pypi_0    pypi

cc @ngimel