Repository: pytorch
Issue: Convolution2D may have a numerical error.
Labels: module: numerical-stability,triaged

## 🐛 Bug

Convolution should be a linear function, therefore it should behave like a linear system where

```
x1, x2 are Conv2D inputs,
delta_x = x1-x2
conv(delta_x) = conv(x1)-conv(x2)
```

However, this is not reproducible by PyTorch.

<!-- A clear and concise description of what the bug is. -->

## To Reproduce

Steps to reproduce the behavior:

1. Make convolution deterministic:
    ```python
    import torch
    import random
    import numpy as np
    
    torch.manual_seed(0)
    random.seed(0)
    np.random.seed(0)
    torch.backends.cudnn.deterministic = True
    ```

2. Set up a simple convolution with 5 input channels and 3 kernels, with size (5, 5), without bias:

     ```python
     conv2d = torch.nn.Conv2d(5, 3, 5, padding=2, bias=False).eval()
     ```

3. Set up the inputs and run convolution
    ```python
     input1 = torch.randn((1, 5, 128, 128))
     input2 = torch.randn((1, 5, 128, 128))
     delta = input1 - input2
     out1 = conv2d(input1)
     out2 = conv2d(input2)
     outd = conv2d(delta)
     ```

4. Since convolution is a linear operation, then `outd = out1-out2`, means `outd-out1+out2=0`, However,
    ```python
    print("out1-out2=", torch.sum(out1-out2))
    print("outd=", torch.sum(outd))
    print("outd-out1+out2=", torch.sum(outd - out1 + out2))
    ```
    outputs
    ```
    out1-out2= tensor(-17.57532501220703125000, grad_fn=<SumBackward0>)
    outd= tensor(-17.57520294189453125000, grad_fn=<SumBackward0>)
    outd-out1+out2= tensor(    0.00009200989734381437, grad_fn=<SumBackward0>)
    ```
    __The last item is not 0! it's about 1e-5.__

1e-5 is way too large to quantify as a rounding error. I'm not very sure what causes this result.

## Expected behavior

The expected behavior of the example above is `outd-out1+out2=0`.

Actually, the following combinations work:

```python
input1 = torch.randint(1, 2, (1, 5, 128, 128))  # all 1s
input2 = torch.randint(1, 2, (1, 5, 128, 128))  # all 1s
delta = input1-input2  # all 0s
```
```python
input1 = torch.randint(1, 2, (1, 5, 128, 128))  # all 1s
input2 = torch.randint(2, 3, (1, 5, 128, 128))  # all 2s
delta = input1-input2  # all -1s
```
```python
input1 = torch.randint(2, 3, (1, 5, 128, 128))  # all 2s
input2 = torch.randint(1, 2, (1, 5, 128, 128))  # all 2s
delta = input1-input2  # all 1s
```

__However, the following combination does not work__:
```python
input1 = torch.randint(1, 3, (1, 5, 128, 128))  # 1 or 2s
input2 = torch.randint(1, 3, (1, 5, 128, 128))  # 1 or 2s
delta = input1-input2  # -1, 0, 1s
```


## Environment

This is produced with PyTorch version 1.7.1.
__The bug is also reproducible on another machine with Pytorch 1.8.1.__

PyTorch version: 1.7.1+cu101
Is debug build: False
CUDA used to build PyTorch: 10.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 16.04.7 LTS (x86_64)
GCC version: (Ubuntu 5.5.0-12ubuntu1~16.04) 5.5.0 20171010
Clang version: Could not collect
CMake version: version 3.14.5

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 10.1.243
GPU models and configuration: GPU 0: GeForce GTX 980 Ti
Nvidia driver version: 418.56
cuDNN version: /usr/local/cuda-10.1/targets/x86_64-linux/lib/libcudnn.so.7
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.5
[pip3] pytorch-lightning==1.1.8
[pip3] torch==1.7.1+cu101
[pip3] torchaudio==0.7.2
[pip3] torchvision==0.8.2+cu101
[conda] numpy                     1.19.5                   pypi_0    pypi
[conda] pytorch-lightning         1.1.8                    pypi_0    pypi
[conda] torch                     1.7.1+cu101              pypi_0    pypi
[conda] torchaudio                0.7.2                    pypi_0    pypi
[conda] torchvision               0.8.2+cu101              pypi_0    pypi


## Additional context

<!-- Add any other context about the problem here. -->