Repository: pytorch
Issue: Conv3d with specific kernel size outputs inconsistent results between FP16 and FP32 in V100 GPU
Labels: module: dependency bug,module: cudnn,module: cuda,module: convolution,triaged

## 🐛 Bug

The Conv3d output inconsistent results between float mode and half mode, if the GPU is V100 and the conv kernel size is (3,1,1) and the input/output channel number are multiple of 8.
I could repro this in PyTorch 1.2.0~PyTorch 1.5.1 with GPU V100-SXM2/PCIE 16GB/32GB

## To Reproduce

repro code
```
def diff_conv_half():
    import torch
    for ch_in in range(2, 256):
        for ch_out in range(2, 256):
            conv = torch.nn.Conv3d(ch_in, ch_out, kernel_size=(3, 1, 1), bias=False, padding=(1, 0, 0)).cuda().half()
            x = torch.rand(1, ch_in, 5, 7, 13).cuda().half()
            y_half = conv(x)

            conv = conv.float()
            x = x.float()
            y_full = conv(x)

            similarity = torch.nn.functional.cosine_similarity(y_half.float().view(1,-1), y_full.view(1,-1)).item()
            if (similarity < 1.0 - 1e-5):
                print("{0}\t{1}\t{2:.4f}".format(ch_in, ch_out, similarity))
```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior

A part of the output looks like below. We can see, if the input channel number and the output channel number are multiple of 8, the cosine similarity of half mode and float mode is not 1.0. 
```
8	8	0.8186
8	16	0.8308
8	24	0.7656
8	32	0.7553
8	40	0.7876
8	48	0.7515
8	56	0.7431
8	64	0.7942
...	...	...
16	8	0.6826
16	16	0.8344
16	24	0.8015
16	32	0.7959
16	40	0.8219
16	48	0.7528
16	56	0.7625
16	64	0.8010
...	...	...
```

<!-- A clear and concise description of what you expected to happen. -->

## Environment

PyTorch version: 1.5.1
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Ubuntu 16.04.5 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609
CMake version: version 3.5.1

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration: GPU 0: Tesla V100-PCIE-32GB
Nvidia driver version: 418.87.00
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.4.1

Versions of relevant libraries:
[pip] numpy==1.16.4
[pip] torch==1.5.1
[pip] torchvision==0.6.0a0+35d732a
[conda] _pytorch_select           0.2                       gpu_0
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               10.1.243             h6bb024c_0
[conda] magma-cuda100             2.5.0                         1    pytorch
[conda] mkl                       2019.4                      243
[conda] mkl-include               2019.4                      243
[conda] mkl-service               2.0.2            py36h7b6447c_0
[conda] mkl_fft                   1.0.12           py36ha843d7b_0
[conda] mkl_random                1.0.2            py36hd81dba3_0
[conda] numpy                     1.16.4           py36h7e9f1db_0
[conda] numpy-base                1.16.4           py36hde5b4d6_0
[conda] pytorch                   1.5.1           py3.6_cuda10.1.243_cudnn7.6.3_0    pytorch
[conda] torchvision               0.6.1                py36_cu101    pytorch



cc @ngimel @csarofeen @ptrblck