Repository: pytorch
Issue: InlineAutodiffSubgraphs in JIT inlines non-differentiable custom groups unexpectedly.
Labels: oncall: jit,module: custom-operators,actionable

## 🐛 Bug

## Issue description

It appears that the `InlineAutodiffSubgraphs` pass inlines non-differentiable custom groups unexpectedly.  As a result, `.backward(...)` cannot be executed on the output from the forward computation.

To fix this issue, `canRunWithAutograd` in `torch/csrc/jit/passes/inline_autodiff_subgraphs.cpp` needs to be updated as shown below:

<pre><code>bool canRunWithAutograd(Node* node) {
  return (node->kind().is_prim() && node->kind() != prim::FusionGroup) ||
      node->kind().is_aten();
}</code></pre>

## Code example

Consider the following code example:
<pre><code>model = torchvision.models.resnet18().eval().to(dtype=torch.float, device='cuda')
x = torch.randn(2, 3, 224, 224, dtype=torch.float, device='cuda', requires_grad=True)
x_ = x.clone().detach().requires_grad_(True)
traced = torch.jit.trace(model, x_, check_trace=False)
out = model(x)
tout = traced(x_)
print(traced.graph_for(x_))
print(out)
print(tout)
out.backward(torch.ones(2, 1000, dtype=float, device='cuda'))
tout.backward(torch.ones(2, 1000, dtype=float, device='cuda'))
print(x.grad)
print(x_.grad)
</code></pre>

Without a custom pass, this works fine and `tout` is printed out as
<pre><code>tensor([[-1.7074,  0.4159,  1.1330,  ...,  1.2971, -0.7362,  0.6159],
        [-1.7468,  0.4392,  1.1862,  ...,  1.1875, -0.6431,  0.6839]],
       device='cuda:0', grad_fn=&lt;DifferentiableGraphBackward&gt;)
</code></pre>

But with a custom pass enabled, I get the error
<pre><code>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</code></pre>

and `tout` is printed out without `grad_fn` as
<pre><code>tensor([[ 0.2352,  0.9813, -0.5402,  ..., -1.0934,  0.7645, -1.1978],
        [ 0.4200,  0.8891, -0.5688,  ..., -1.0392,  0.7479, -1.2435]],
       device='cuda:0')
</code></pre>

With the custom pass, one of the two `prim::DifferentiableGraph` is inlined in the graph making custom groups appear as differentiable groups.

## System Info
PyTorch version: 1.3.0a0+ee77ccb
Is debug build: No
CUDA used to build PyTorch: 10.1.243

OS: Ubuntu 18.04.4 LTS
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
CMake version: version 3.14.0

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: Quadro GV100
Nvidia driver version: 440.33.01
cuDNN version: Could not collect

Versions of relevant libraries:
[pip] numpy==1.16.4
[pip] torch==1.3.0a0+ee77ccb
[pip] torchvision==0.4.2
[conda] blas                      1.0                         mkl
[conda] mkl                       2019.4                      243
[conda] mkl-include               2019.4                      243
[conda] mkl-service               2.3.0            py37he904b0f_0
[conda] mkl_fft                   1.0.15           py37ha843d7b_0
[conda] mkl_random                1.1.0            py37hd6b4f25_0
[conda] torch                     1.3.0a0+ee77ccb           dev_0    <develop>
[conda] torchvision               0.4.2                    pypi_0    pypi



cc @suo