Repository: pytorch
Issue: [BUG] found a weird problem of nn.conv2d in pytorch 2.2.0
Labels: module: numerical-stability,module: cudnn,triaged

### 🐛 Describe the bug

I got weird result of nn.conv2d in T4 GPU
the conv2d result seems wrong in different batch size.

I could understand different shape maybe run different kernel, but the diff seems too large for float16.
BTW, in the attachment I found it seems run the same kernel

so why the result have this large diffs

before I run the script, I ran :
export CUDA_VISIBLE_DEVICES=3

## code 

```python
import torch
import torch.nn as nn
import nvtx

loaded_conv_layer = nn.Conv2d(320, 640, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
# Load the state_dict into the new Conv2d layer
loaded_conv_layer.load_state_dict(torch.load("conv_model.pth"))
# Move to CUDA if required
loaded_conv_layer = loaded_conv_layer.to(torch.float16).to("cuda:0")

conv_not_same = nvtx.start_range(message="conv_not_same", color="green")
testa = torch.ones((4,320,48,48)).to(torch.float16).to("cuda:0")
testb = torch.ones((2,320,48,48)).to(torch.float16).to("cuda:0")
ra = loaded_conv_layer(testa)
rb = loaded_conv_layer(testb)
approx_equal = torch.allclose(rb, ra[0:2], rtol=1e-05, atol=1e-08)
print(f"conv result not same", approx_equal)
abs_diff = torch.abs(rb - ra[0:2])
max_diff = abs_diff.max()
mean_diff = abs_diff.mean()
print("Max Absolute Difference:", max_diff.item())
print("Mean Absolute Difference:", mean_diff.item())
nvtx.end_range(conv_not_same)
print(" <============>")
conv_nearly_same = nvtx.start_range(message="conv_nearly_same", color="red")
testa = torch.ones((4,320,48,48)).to(torch.float16).to("cuda:0")
testa.fill_(0.1)
testb = torch.ones((2,320,48,48)).to(torch.float16).to("cuda:0")
testb.fill_(0.1)
ra = loaded_conv_layer(testa)
rb = loaded_conv_layer(testb)
approx_equal = torch.allclose(rb, ra[0:2], rtol=1e-05, atol=1e-08)
print(f"conv result nearly same", approx_equal)
abs_diff = torch.abs(rb - ra[0:2])
max_diff = abs_diff.max()
mean_diff = abs_diff.mean()
print("Max Absolute Difference:", max_diff.item())
print("Mean Absolute Difference:", mean_diff.item())
print(" <============>")
nvtx.end_range(conv_nearly_same)

conv_zero_same = nvtx.start_range(message="conv_zero_same", color="green")
testa = torch.ones((4,320,48,48)).to(torch.float16).to("cuda:0")
testa.fill_(0)
testb = torch.ones((2,320,48,48)).to(torch.float16).to("cuda:0")
testb.fill_(0)
ra = loaded_conv_layer(testa)
rb = loaded_conv_layer(testb)
approx_equal = torch.allclose(rb, ra[0:2], rtol=1e-05, atol=1e-08)
print(f"conv result zero same", approx_equal)
abs_diff = torch.abs(rb - ra[0:2])
max_diff = abs_diff.max()
mean_diff = abs_diff.mean()
print("Max Absolute Difference:", max_diff.item())
print("Mean Absolute Difference:", mean_diff.item())
print(" <============>")
nvtx.end_range(conv_zero_same)

conv_same = nvtx.start_range(message="conv_same", color="blue")
testa = torch.ones((4,320,16,16)).to(torch.float16).to("cuda:0")
testb = torch.ones((2,320,16,16)).to(torch.float16).to("cuda:0")
ra = loaded_conv_layer(testa)
rb = loaded_conv_layer(testb)
approx_equal = torch.allclose(rb, ra[0:2], rtol=1e-05, atol=1e-08)
print(f"conv result same", approx_equal)
abs_diff = torch.abs(rb - ra[0:2])
max_diff = abs_diff.max()
mean_diff = abs_diff.mean()
print("Max Absolute Difference:", max_diff.item())
print("Mean Absolute Difference:", mean_diff.item())
nvtx.end_range(conv_same)

```
## result

```
conv result not same False
Max Absolute Difference: 0.03125
Mean Absolute Difference: 1.0728836059570312e-06
 <============>
conv result nearly same False
Max Absolute Difference: 0.00390625
Mean Absolute Difference: 1.6093254089355469e-06
 <============>
conv result zero same True
Max Absolute Difference: 0.0
Mean Absolute Difference: 0.0
 <============>
conv result same True
Max Absolute Difference: 0.0
Mean Absolute Difference: 0.0

```
and here is the nsys result and the pth

[nsys-and-pth.zip](https://github.com/user-attachments/files/16843292/nsys-and-pth.zip)


### Versions



## versions

```python
pip list | grep torch
  efficientnet-pytorch      0.7.1
  open-clip-torch           2.24.0
  torch                     2.2.0
  torchmetrics              1.4.0.post0
  torchsde                  0.2.6
  torchvision               0.17.0
```
## hardware

```bash
 nvidia-smi
Tue Sep  3 04:02:23 2024       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.47.03    Driver Version: 510.47.03    CUDA Version: 12.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            On   | 00000000:12:00.0 Off |                    0 |
| N/A   74C    P0    31W /  70W |  11962MiB / 15360MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla T4            On   | 00000000:13:00.0 Off |                    0 |
| N/A   56C    P8    16W /  70W |      2MiB / 15360MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  Tesla T4            On   | 00000000:37:00.0 Off |                    0 |
| N/A   60C    P8    17W /  70W |      2MiB / 15360MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  Tesla T4            On   | 00000000:86:00.0 Off |                    0 |
| N/A   58C    P8    16W /  70W |      2MiB / 15360MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
```





cc @csarofeen @ptrblck @xwang233