Repository: pytorch
Issue: Spurious negative output in convolution of positive tensors
Labels: module: docs,module: convolution,triaged

The gist of the issue is that at one point in my network in my project I feed positive float32 inputs `X` and `W` into a 1d (or 2d) convolution, but unexpectedly obtain explicitly negative outputs.

## Details
The X input is nonnegative about 1e-2 with many explicit 0 (after an elementwise square, or abs), and the W input is positive but with quite large range of [1e–8, 1e+8] (after an exp). I take `F.conv2d(X, W, None)` and **expect nonnegative** outputs (finite or infinite), but actually get **significantly negative** values. The downstream computations rely on the result of this operation to be at least nonnegative.

I tried `torch.backends.cudnn.deterministic = True`, or fp64, or clamping the `W` value and each time the problem seemed to had gone away. In my project I ended up clamping the result of the convolution to nonnegative range, and it sufficed.

Having observed this oddity got me worried, since in my understanding of floating point arithmetic the loss of precision and round off could yield imprecise results, but not **flip the sign** of the overall result. I read in the docs, that non-deterministic cudnn algorithms might induce this behaviour and yield inconsistent results, but I did not expect that atomic additions (according to cudnn reference) could spuriously flip the sign bit of a 32bit float.

I understand that Github issues are not a Q&A support forum, but I think that this inconsistency, might of interest, and would very much appreciate any discussion of guarantees of single precision operations.

## Minimally reproducing example
I came up with a minimal example of this odd behaviour. My observations are as follows (based on this snippet, and on the real data within the network in my project):
* negative outputs of various magnitude occur **only if** deterministic is *False*, and data is *fp32*, and the input `X` is a moderately large-dim tensor.
* `X` and `W` of dims `...x12x12` and `...x4x4`, respectively, give a lot of very small fp32 numbers, some of which are negative
* `X` and `W` of dims `...x11x11` and `...x3x3`, respectively, produce very rare negative tiny fp32, but mostly expected zeros

These suggest that some optimizations within cudnn might be responsible.

The following code constructs explicitly nonnegative `X` and `W`: `X` all zeros except for a single **1** at the *centre*, `W` is a 4x4 kernel with constant **1e-3**. I attach the pickled  [result.gz](https://github.com/pytorch/pytorch/files/3935238/result.gz)

```python
import gzip
import torch
import torch.nn.functional as F

# change these
torch.backends.cudnn.deterministic = False
devtype = dict(device=torch.device("cuda:0"), dtype=torch.float32)

# x -- zeros with a spike
x = torch.zeros(1024, 1, 12, 12).to(**devtype)  
x[..., 5, 5] = 1.

# small nonnegative values
w = (torch.ones(1, 1, 4, 4) * 1e-3).to(**devtype)
assert x.ge(0).all() and w.ge(0).all()

# conv1d/2d
result = F.conv2d(x, w, None)
with gzip.open("result.gz", "wb") as fout:
    torch.save(dict(x=x, w=w, result=result), fout)

assert result.ge(0).all()
```

To unpack the attached `.gz` pickled result.
```python
import gzip
import torch

with gzip.open("result.gz", "rb") as fin:
    pack = torch.load(fin)

pack["result]
```

Here is a link to a [colab notebook](https://colab.research.google.com/drive/136n8oKDitchxEkXMhV7AJeYfFUayiS4X), where it can be seen, that the output is spurious even for `X` is `25 x 1 x 12 x 12`. Especially striking is that `24 x 1 x 12 x 12` appears to be correct.

## Specs and versions
Below are very abridged specs:

`Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz, 256GB RAM`, `4x GeForce GTX 1080 Ti` with `Driver Version: 390.77` on `Linux Mint 18.3 Sylvia (GNU/Linux 4.10.0-38-generic x86_64)` and conda `python 3.7.4` packages:
* `pytorch                   1.1.0           py3.7_cuda9.0.176_cudnn7.5.1_0    pytorch`
* `cudatoolkit               9.0                  h13b8566_0`


cc @jlin27 @mruberry