Repository: pytorch
Issue: F.conv2d double precision error
Labels: needs reproduction,module: nn,module: convolution,triaged

### 🐛 Describe the bug

I have resnet18 network, and I'm rewritting all the nn.Conv2d forward to F.conv2d like below:

```python
def forward(self, input):

    out = F.conv2d(
        input.to(torch.double), # use double because all my input and weight are int
        self.weight.to(torch.double),
        None,
        self.stride,
        self.padding,
        self.dilation,
        self.groups,
    )
    
    return out
```


after rewritting this, I can see a huge accuracy degration in the middle of the network, like the layer2_0_conv2 layer below, the cos sim is low and mre is high, compared to original fp32 implementation.
```
NAME                     COSSIM      MRE    RE[0-1)    RE[1-10)    RE[10-50)    RE[50-100)    RE[100-inf)
---------------------  --------  -------  ---------  ----------  -----------  ------------  -------------
x                        1.0050   0.5604    69.9136     27.5637       1.4453        0.3696         0.7078
conv1                    1.0218   1.7797    51.3948     44.4813       3.4379        0.1530         0.5330
layer1_0_conv1           1.0006   3.2676    52.6043     38.2593       7.4568        0.4410         1.2386
layer1_0_conv2           1.0028   5.4713    10.5648     54.6322      25.0674        1.9363         7.7992
add                      1.0015   3.3648    33.6626     53.1951      10.9462        0.6774         1.5186
layer1_1_conv1           0.9989   6.5470    57.4115     25.3315      13.4682        1.3461         2.4426
layer1_1_conv2           1.0014   9.4256     6.8843     43.5271      34.6559        3.7495        11.1831
add_1                    1.0004   5.8129    31.7068     44.4226      18.9624        1.7789         3.1293
layer2_0_conv1           0.8877  15.3905    60.2232     18.9320      12.2579        2.3846         6.2023
layer2_0_conv2           0.5294  33.1446     6.9361     43.1119      33.5667        3.8686        12.5166
layer2_0_downsample_0    0.9975   9.1667     7.2486     45.5902      33.8947        3.8256         9.4409
add_2                    0.6718  24.5588    50.7829     23.7198      16.9484        2.3963         6.1525
layer2_1_conv1           0.3974  63.9974    75.0178      9.3626       8.9190        1.5931         5.1075
layer2_1_conv2           0.4271  49.5895     6.3084     42.1205      34.8103        3.4991        13.2618
add_3                    0.5568  36.0842    50.7192     20.9595      18.2571        2.5221         7.5421
layer3_0_conv1           0.6939  27.0735    65.6461     15.1996      12.0083        1.8693         5.2767
layer3_0_conv2           0.6208  30.0869     7.0117     44.5205      33.2517        3.3924        11.8238
layer3_0_downsample_0    0.6625  27.9017     6.8350     44.8574      33.1326        3.2313        11.9437
add_4                    0.7645  22.0159    55.9957     20.7157      15.5012        2.0999         5.6875
layer3_1_conv1           0.7387  25.5691    79.3183      8.5265       7.4659        1.1945         3.4949
layer3_1_conv2           0.7267  25.6001     7.6617     47.3350      31.1598        2.8151        11.0285
add_5                    0.8525  18.9333    57.3849     18.5594      15.9140        2.0720         6.0696
layer4_0_conv1           0.9248  18.2137    80.9318      7.1765       7.1802        1.2746         3.4369
layer4_0_conv2           0.9095  15.8882     7.2577     45.5770      33.2077        3.7004        10.2572
layer4_0_downsample_0    0.8635  16.9231     8.1987     50.5042      30.2375        3.0385         8.0211
add_6                    0.9333  19.3010    79.8709      7.1314       7.6642        1.3807         3.9529
layer4_1_conv1           0.9700  15.5810    80.3773      7.5394       7.3790        1.2851         3.4192
layer4_1_conv2           0.9680  15.1906     5.1203     36.8358      38.9180        5.0832        14.0427
add_7                    0.9767  13.9611    47.5788     21.2903      19.8535        2.8720         8.4054
fc                       0.9695  64.1238     0.0648      0.5008       7.8328       77.0863        14.5152
```

but if I change forward like below, input and weight are set to torch.float32, the accuracy is good as expected.

```
def forward(self, input):

    out = F.conv2d(
        input.to(torch.float32),
        self.weight.to(torch.float32),
        None,
        self.stride,
        self.padding,
        self.dilation,
        self.groups,
    )
    
    return out
```

the result is as below
```
NAME                     COSSIM      MRE    RE[0-1)    RE[1-10)    RE[10-50)    RE[50-100)    RE[100-inf)
x                        1.0050   0.5604    69.9136     27.5637       1.4453        0.3696         0.7078
conv1                    1.0400   1.7797    51.3948     44.4813       3.4379        0.1693         0.5168
layer1_0_conv1           1.0004   3.2676    52.6043     38.2593       7.4568        0.4776         1.2020
layer1_0_conv2           1.0082   5.4713    10.5648     54.6321      25.0675        2.4157         7.3198
add                      1.0040   3.3649    33.6626     53.1950      10.9463        0.7463         1.4498
layer1_1_conv1           0.9992   6.5470    57.4116     25.3315      13.4682        1.4206         2.3681
layer1_1_conv2           1.0028   9.4256     6.8844     43.5272      34.6557        4.2997        10.6329
add_1                    1.0027   5.8129    31.7069     44.4224      18.9624        1.8921         3.0163
layer2_0_conv1           0.9972   8.7699    60.8285     19.4307      12.4219        2.3963         4.9226
layer2_0_conv2           0.9964   9.3425     7.1345     44.3354      34.3226        4.2637         9.9437
layer2_0_downsample_0    1.0015   9.1666     7.2488     45.5899      33.8948        4.2125         9.0540
add_2                    0.9968   8.8672    51.4047     24.3706      17.2629        2.4777         4.4841
layer2_1_conv1           0.9944  11.6035    75.7740      9.6336       9.1449        1.7508         3.6967
layer2_1_conv2           1.0016   9.6751     6.4960     43.3523      35.7021        4.0670        10.3825
add_3                    0.9956  10.2795    51.3361     21.5703      18.7162        2.7495         5.6279
layer3_0_conv1           0.9953   9.5920    66.1836     15.6372      12.3145        2.0208         3.8439
layer3_0_conv2           0.9968   8.9332     7.2199     45.8058      34.0482        3.7997         9.1264
layer3_0_downsample_0    0.9962   8.9033     7.0339     46.1205      33.9341        3.6753         9.2362
add_4                    0.9957   9.0903    56.5298     21.3055      15.8756        2.2368         4.0524
layer3_1_conv1           0.9948  10.4419    79.7235      8.7571       7.6487        1.2972         2.5734
layer3_1_conv2           0.9984   7.5824     7.8643     48.6204      31.9571        3.2239         8.3343
add_5                    0.9950  10.1163    57.7520     19.1045      16.3208        2.2451         4.5776
layer4_0_conv1           0.9943  11.5994    81.1597      7.3895       7.3724        1.3899         2.6885
layer4_0_conv2           0.9960   8.5710     7.4612     46.8369      33.8351        3.8244         8.0423
layer4_0_downsample_0    0.9967   7.9210     8.4271     51.8674      30.8565        3.0739         5.7750
add_6                    0.9939  12.3478    80.1981      7.3456       7.8742        1.5042         3.0779
layer4_1_conv1           0.9939  11.8232    80.5118      7.7524       7.6049        1.3902         2.7406
layer4_1_conv2           0.9961  11.3823     5.2485     37.7790      39.6999        5.3027        11.9699
add_7                    0.9959  10.1819    48.2949     21.8521      20.2419        2.9519         6.6592
fc                       0.9952  62.9202     0.0539      0.4617       7.7418       78.5309        13.2117
```

I was wondering if there are some bugs for F.conv2d double precision, is it an accumulative error?

### Versions

PyTorch version: 2.1.2+cu118
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.10.2
Libc version: glibc-2.27

Python version: 3.8.18 (default, Sep 11 2023, 13:40:15)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-3.10.0-1160.99.1.el7.x86_64-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: 12.1.105
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA A100-PCIE-40GB
Nvidia driver version: 535.129.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.0.4
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn.so.8.8.1
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.8.1
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.8.1
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.8.1
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.8.1
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.8.1
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.8.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              80
On-line CPU(s) list: 0-79
Thread(s) per core:  2
Core(s) per socket:  20
Socket(s):           2
NUMA node(s):        2
Vendor ID:           GenuineIntel
CPU family:          6
Model:               85
Model name:          Intel Xeon Processor (Skylake, IBRS)
Stepping:            4
CPU MHz:             2992.865
BogoMIPS:            5985.73
L1d cache:           32K
L1i cache:           32K
L2 cache:            4096K
L3 cache:            16384K
NUMA node0 CPU(s):   0-39
NUMA node1 CPU(s):   40-79
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single rsb_ctxsw ibrs ibpb fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 arat spec_ctrl

Versions of relevant libraries:
[pip3] espnet-onnx==0.2.0
[pip3] facenet-pytorch==2.5.3
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.23.1
[pip3] onnx==1.15.0
[pip3] onnx-graphsurgeon==0.3.27
[pip3] onnx-simplifier==0.4.35
[pip3] onnxruntime==1.16.3
[pip3] onnxruntime-extensions==0.9.0
[pip3] onnxsim==0.4.35
[pip3] onnxslim==0.1.16
[pip3] torch==2.1.2+cu118
[pip3] torch-complex==0.4.3
[pip3] torchaudio==2.1.2+cu118
[pip3] torchvision==0.16.2+cu118
[pip3] triton==2.1.0
[conda] facenet-pytorch           2.5.3                    pypi_0    pypi
[conda] numpy                     1.23.1                   pypi_0    pypi
[conda] torch                     2.1.2+cu118              pypi_0    pypi
[conda] torch-complex             0.4.3                    pypi_0    pypi
[conda] torchaudio                2.1.2+cu118              pypi_0    pypi
[conda] torchvision               0.16.2+cu118             pypi_0    pypi
[conda] triton                    2.1.0                    pypi_0    pypi

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki