Repository: pytorch
Issue: [Torchscript] some activations backward are not fused when used with linear
Labels: oncall: jit

### 🐛 Describe the bug

## Description
The backward of some activation functions (e.g. `SiLU`, `erf`) contain many kernels, and there will be more kernels if the user requires higher order derivatives (e.g. Physics Informed Neural Networks in Modulus).
For the activation functions that have symbolic gradient defined at [symbolic_script.cpp](https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/runtime/symbolic_script.cpp), their backward are not fused when used with linear.

For example:
The backward of the following standalone `torch.erf()` could be fused.
```py
class Erf(torch.nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        return torch.erf(x)
```

But when used with linear, the backward of these activation functions is not fused anymore. And the performance of the scripted module is even worse than the eager mode.

```py
linear_erf = torch.nn.Sequential(
    torch.nn.Linear(512, 512),
    Erf(),
    torch.nn.Linear(512, 512),
    Erf(),
    torch.nn.Linear(512, 512),
    Erf(),
    torch.nn.Linear(512, 512),
)
```

After some debugging, we found this happens when the DifferentiableGraph outputs contain an alias of the inputs.
And [unmergeOutputsAlisingInputs](https://github.com/pytorch/pytorch/blob/1bbea3c3a2ed0843de1bfdd360b999ee21cee635/torch/csrc/jit/passes/utils/subgraph_utils.cpp#L435) might be the reason that torchscript decided to unfuse the DifferentiableGraph.

For more detail:
The unfused DifferentiableGraph contains the calculation of: linear_grad_input + erf_backward + linear_grad_input + erf_backward + linear_grad_input ...
This graph returns an alias of the grad_output, because it is needed to calculate the linear_grad_weight.

## Steps To Reproduce: 
benchmark script: https://gist.github.com/yueyericardo/0d89a3a74c874c68a5a8729891a459a8#file-test_linear_erf-py

Sample outputs
benchmark:
```
$ python test_linear_erf.py 
erf                                : 1.903 ms/step
erf_scripted                       : 0.939 ms/step
linear_erf                         : 16.079 ms/step
linear_erf_scripted                : 16.801 ms/step  # not fused
```
profile the kernels, logs: https://gist.github.com/yueyericardo/0d89a3a74c874c68a5a8729891a459a8#file-linear_erf_profile-log
```
$ python test_linear_erf.py -p 
```

The command I used for JIT graph debugging:
```
PYTORCH_JIT_LOG_LEVEL=">>>profiling_graph_executor_impl:>>>create_autodiff_subgraphs:>>>subgraph_utils" python test_linear_erf.py 
```

### Versions

```
python collect_env.py
Collecting environment information...
PyTorch version: 1.13.0a0+340c412
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.23.2
Libc version: glibc-2.31

Python version: 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:10)  [GCC 10.3.0] (64-bit runtime)
Python platform: Linux-5.4.0-91-generic-x86_64-with-glibc2.10
Is CUDA available: True
CUDA runtime version: 11.7.99
GPU models and configuration: GPU 0: NVIDIA A100 80GB PCIe
Nvidia driver version: 510.73.08
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.22.4
[pip3] pytorch-quantization==2.1.2
[pip3] torch==1.13.0a0+340c412
[pip3] torch-tensorrt==1.1.0a0
[pip3] torchtext==0.13.0a0
[pip3] torchvision==0.13.0a0
[conda] mkl                       2020.4             h726a3e6_304    conda-forge
[conda] mkl-include               2020.4             h726a3e6_304    conda-forge
[conda] numpy                     1.22.4           py38h99721a1_0    conda-forge
[conda] pytorch-quantization      2.1.2                    pypi_0    pypi
[conda] torch                     1.13.0a0+340c412          pypi_0    pypi
[conda] torch-tensorrt            1.1.0a0                  pypi_0    pypi
[conda] torchtext                 0.13.0a0                 pypi_0    pypi
[conda] torchvision               0.13.0a0                 pypi_0    pypi
```