Repository: pytorch
Issue: nn.BatchNorm2d layers cause redundant Identity operators in exported ONNX
Labels: module: onnx,triaged

### 🐛 Describe the bug

I'm working on a Siamese Network that accepts two input images.

This network mainly has two blocks: a down-sampling block (4 conv), and a block of 6 residuals, and it's being used in two branches, one operates on the first image, and the other one operates on the second image.

However, when exporting to onnx via `torch.onnx.export` API, `nn.BatchNorm2d` causes the conv operators to have redundant Identity operators.

Code to reproduce:

``` 
import torch
from torch import nn


class ResBlock(nn.Module):
    def __init__(self, in_channel, out_channel, dilation=1, stride=1, downsample=None):
        super(ResBlock, self).__init__()

        padding = dilation

        self.conv1 = nn.Conv2d(in_channel, out_channel, kernel_size=3, stride=stride,
                     padding=padding, dilation=dilation, bias=False)
        self.bn1 = nn.BatchNorm2d(out_channel)
        self.relu1 = nn.LeakyReLU(negative_slope=0.2, inplace=True)

        self.conv2 = nn.Conv2d(out_channel, out_channel, kernel_size=3, stride=stride,
                               padding=padding, dilation=dilation, bias=False)
        self.bn2 = nn.BatchNorm2d(out_channel)
        self.relu2 = nn.LeakyReLU(negative_slope=0.2, inplace=True)

        self.downsample = downsample
        self.stride = stride
        self.in_ch = in_channel
        self.out_ch = out_channel
        self.p = padding
        self.d = dilation

    def forward(self, x):
        residual = x

        out = self.conv1(x)
        out = self.bn1(out)

        out = self.relu1(out)
        out = self.conv2(out)
        out = self.bn2(out)

        out += residual

        out = self.relu2(out)

        return out


class TmpModel(nn.Module):
    def __init__(self):
        super(TmpModel, self).__init__()

        self.down_sampling = nn.Sequential(
            nn.Conv2d(3, 32, 5, stride=2, padding=2),
            nn.Conv2d(32, 32, 5, stride=2, padding=2),
            nn.Conv2d(32, 32, 5, stride=2, padding=2),
            nn.Conv2d(32, 32, 5, stride=2, padding=2),
        )

        self.res = nn.Sequential(
            ResBlock(32, 32),
            ResBlock(32, 32),
            ResBlock(32, 32),
            ResBlock(32, 32),
            ResBlock(32, 32),
            ResBlock(32, 32),
            nn.Conv2d(32, 32, 3, 1, 1),
        )

    def forward_once_1(self, x):
        output = self.down_sampling(x)
        output = self.res(output)

        return output

    def forward(self, x, y):
        output_l = self.forward_once_1(x)
        output_r = self.forward_once_1(y)

        return output_l + output_r


if __name__ == "__main__":
    model = TmpModel()
    model.eval()

    x = torch.rand(1, 3, 368, 1232)
    y = torch.rand(1, 3, 368, 1232)

    _ = model(x, y)

    torch.onnx.export(
        model,
        args=(x, y),
        f="./tmp.onnx",
        do_constant_folding=True,
        input_names=["input_1", "input_2"],
        output_names=["output"],
        opset_version=14,
        export_params=True,
    )
```

![redundant_identity_ops](https://github.com/pytorch/pytorch/assets/63457095/029c55d4-5713-48be-9a4d-5b5ece33a80f)


### Versions

Collecting environment information...
PyTorch version: 2.1.1
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 Enterprise
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.8.18 (default, Sep 11 2023, 13:39:12) [MSC v.1916 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19045-SP0
Is CUDA available: True
CUDA runtime version: 11.2.67
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: Quadro P2000
Nvidia driver version: 460.89
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture=9
CurrentClockSpeed=2592
DeviceID=CPU0
Family=198
L2CacheSize=1536
L2CacheSpeed=
Manufacturer=GenuineIntel
MaxClockSpeed=2592
Name=Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
ProcessorType=3
Revision=

Versions of relevant libraries:
[pip3] numpy==1.24.3
[pip3] onnx==1.13.1
[pip3] torch==2.1.1
[pip3] torchaudio==2.1.1
[pip3] torchvision==0.16.1
[conda] blas                      1.0                         mkl
[conda] mkl                       2023.1.0         h6b88ed4_46358
[conda] mkl-service               2.4.0            py38h2bbff1b_1
[conda] mkl_fft                   1.3.8            py38h2bbff1b_0
[conda] mkl_random                1.2.4            py38h59b6b97_0
[conda] numpy                     1.24.3           py38h79a8e48_1
[conda] numpy-base                1.24.3           py38h8a87ada_1
[conda] pytorch                   2.1.1           py3.8_cuda11.8_cudnn8_0    pytorch
[conda] pytorch-cuda              11.8                 h24eeafa_5    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                2.1.1                    pypi_0    pypi
[conda] torchvision               0.16.1                   pypi_0    pypi