Repository: pytorch
Issue: Does torch.utils.checkpoint compatible with torch.cuda.make_graphed_callables?
Labels: module: checkpoint,triaged,module: cuda graphs

### 🐛 Describe the bug

When attempting to use
```
model = torch.cuda.make_graphed_callables(model, (rand_data,))
```
and our model contains checkpoints or sequential checkpoints like this:
```
input_next = torch.utils.checkpoint.checkpoint_sequential(self.layer, segments, input_prev)
```
then we got this error:
```
Traceback (most recent call last):
  File "multitask.py", line 43, in <module>
    main()
  File "multitask.py", line 36, in main
    S.initialize(args)
  File "/mnt/lustre/chendingyu1/multitask-unify-democode/core/solvers/solver_fp16.py", line 63, in initialize
    super().initialize(args)
  File "/mnt/lustre/chendingyu1/multitask-unify-democode/core/solvers/solver.py", line 317, in initialize
    self.create_model()
  File "/mnt/lustre/chendingyu1/multitask-unify-democode/core/solvers/solver_fp16.py", line 109, in create_model
    model = torch.cuda.make_graphed_callables(model, (rand_data,))
  File "/mnt/cache/share/spring/conda_envs/miniconda3/envs/s0.3.5/lib/python3.7/site-packages/torch/cuda/graphs.py", line 264, in make_graphed_callables
    allow_unused=False)
  File "/mnt/cache/share/spring/conda_envs/miniconda3/envs/s0.3.5/lib/python3.7/site-packages/torch/autograd/__init__.py", line 277, in grad
    allow_unused, accumulate_grad=False)  # Calls into the C++ engine to run the backward pass
  File "/mnt/cache/share/spring/conda_envs/miniconda3/envs/s0.3.5/lib/python3.7/site-packages/torch/autograd/function.py", line 253, in apply
    return user_fn(self, *args)
  File "/mnt/cache/share/spring/conda_envs/miniconda3/envs/s0.3.5/lib/python3.7/site-packages/torch/utils/checkpoint.py", line 103, in backward
    "Checkpointing is not compatible with .grad() or when an `inputs` parameter"
RuntimeError: Checkpointing is not compatible with .grad() or when an `inputs` parameter is passed to .backward(). Please use .backward() and do not pass its `inputs` argument.
```
If we don't use checkpoints in our model like
```
input_next = self.layer(input_prev)
```
then we didn't get any errors.

Does checkpoint compatible with torch.cuda.make_graphed_callables?

### Versions

PyTorch version: 1.11.0+cu113
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: CentOS Linux 7 (Core) (x86_64)
GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)
Clang version: Could not collect
CMake version: version 2.8.12.2
Libc version: glibc-2.17

Python version: 3.7.11 (default, Jul 27 2021, 14:32:16)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-3.10.0-957.el7.x86_64-x86_64-with-centos-7.6.1810-Core
Is CUDA available: True
CUDA runtime version: 9.0.176

Nvidia driver version: 460.32.03

Versions of relevant libraries:
[pip3] numpy==1.21.5
[pip3] spring==0.7.2+cu112.torch1110.mvapich2.nartgpu.develop.805601a8
[pip3] torch==1.11.0+cu113
[pip3] torchvision==0.12.0+cu113
[conda] numpy                     1.21.5                   pypi_0    pypi
[conda] spring                    0.7.0+cu112.torch1110.mvapich2.pmi2.nartgpu          pypi_0    pypi
[conda] torch                     1.11.0+cu113             pypi_0    pypi
[conda] torchvision               0.12.0+cu113             pypi_0    pypi

cc @mcarilli @ezyang