Repository: pytorch
Issue: caffe2 `DEPTHWISE3x3.Conv` test is broken
Labels: caffe2,module: tests,triaged

## 🐛 Bug

`DEPTHWISE3x3.Conv` test fails with following lengthy error message:
```
$ ./bin/depthwise3x3_conv_op_test  | tail -n 50
/home/nshulga/git/pytorch-worktree/caffe2/share/contrib/depthwise/depthwise3x3_conv_op_test.cc:144: Failure
Value of: relErr <= maxRelErr || absErr <= absErrForRelErrFailure
  Actual: false
Expected: true
75 70 (rel err 0.068965516984462738) (1 1 91 78) running N 2 inputC 2 H 90 W 84 outputC 2 kernelH 3 kernelW 3 strideH 1 strideW 1 padT 2 padL 2 padB 2 padR 2 group 2
/home/nshulga/git/pytorch-worktree/caffe2/share/contrib/depthwise/depthwise3x3_conv_op_test.cc:144: Failure
Value of: relErr <= maxRelErr || absErr <= absErrForRelErrFailure
  Actual: false
Expected: true
75 70 (rel err 0.068965516984462738) (1 1 91 79) running N 2 inputC 2 H 90 W 84 outputC 2 kernelH 3 kernelW 3 strideH 1 strideW 1 padT 2 padL 2 padB 2 padR 2 group 2
/home/nshulga/git/pytorch-worktree/caffe2/share/contrib/depthwise/depthwise3x3_conv_op_test.cc:144: Failure
Value of: relErr <= maxRelErr || absErr <= absErrForRelErrFailure
  Actual: false
Expected: true
75 70 (rel err 0.068965516984462738) (1 1 91 80) running N 2 inputC 2 H 90 W 84 outputC 2 kernelH 3 kernelW 3 strideH 1 strideW 1 padT 2 padL 2 padB 2 padR 2 group 2
/home/nshulga/git/pytorch-worktree/caffe2/share/contrib/depthwise/depthwise3x3_conv_op_test.cc:144: Failure
Value of: relErr <= maxRelErr || absErr <= absErrForRelErrFailure
  Actual: false
Expected: true
75 70 (rel err 0.068965516984462738) (1 1 91 81) running N 2 inputC 2 H 90 W 84 outputC 2 kernelH 3 kernelW 3 strideH 1 strideW 1 padT 2 padL 2 padB 2 padR 2 group 2
/home/nshulga/git/pytorch-worktree/caffe2/share/contrib/depthwise/depthwise3x3_conv_op_test.cc:144: Failure
Value of: relErr <= maxRelErr || absErr <= absErrForRelErrFailure
  Actual: false
Expected: true
75 70 (rel err 0.068965516984462738) (1 1 91 82) running N 2 inputC 2 H 90 W 84 outputC 2 kernelH 3 kernelW 3 strideH 1 strideW 1 padT 2 padL 2 padB 2 padR 2 group 2
/home/nshulga/git/pytorch-worktree/caffe2/share/contrib/depthwise/depthwise3x3_conv_op_test.cc:144: Failure
Value of: relErr <= maxRelErr || absErr <= absErrForRelErrFailure
  Actual: false
Expected: true
75 70 (rel err 0.068965516984462738) (1 1 91 83) running N 2 inputC 2 H 90 W 84 outputC 2 kernelH 3 kernelW 3 strideH 1 strideW 1 padT 2 padL 2 padB 2 padR 2 group 2
/home/nshulga/git/pytorch-worktree/caffe2/share/contrib/depthwise/depthwise3x3_conv_op_test.cc:144: Failure
Value of: relErr <= maxRelErr || absErr <= absErrForRelErrFailure
  Actual: false
Expected: true
50 45 (rel err 0.10526315867900848) (1 1 91 84) running N 2 inputC 2 H 90 W 84 outputC 2 kernelH 3 kernelW 3 strideH 1 strideW 1 padT 2 padL 2 padB 2 padR 2 group 2
/home/nshulga/git/pytorch-worktree/caffe2/share/contrib/depthwise/depthwise3x3_conv_op_test.cc:144: Failure
Value of: relErr <= maxRelErr || absErr <= absErrForRelErrFailure
  Actual: false
Expected: true
25 20 (rel err 0.2222222238779068) (1 1 91 85) running N 2 inputC 2 H 90 W 84 outputC 2 kernelH 3 kernelW 3 strideH 1 strideW 1 padT 2 padL 2 padB 2 padR 2 group 2
[  FAILED  ] DEPTHWISE3x3.Conv (143 ms)
[----------] 1 test from DEPTHWISE3x3 (143 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (143 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] DEPTHWISE3x3.Conv

 1 FAILED TEST

```
## To Reproduce

1. Checked out pytorch at `b33e38ec475017868534eb114741ad32c9d3b248` on Linux
2. Build it without GPU nor MKL_DNN acceleration (i.e. invoke `-DUSE_CUDA=NO -DUSE_MKLDNN=OFF`
3. Compile and run `depthwise3x3_conv_op_test`

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment
Collecting environment information...
PyTorch version: 1.5.0
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Fedora release 30 (Thirty)
GCC version: (GCC) 9.2.1 20190827 (Red Hat 9.2.1-1)
CMake version: version 3.16.4

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.1.168
GPU models and configuration: GPU 0: GeForce RTX 2080
Nvidia driver version: 440.31
cuDNN version: /usr/lib64/libcudnn.so.7.5.0

Versions of relevant libraries:
[pip3] numpy==1.16.4
[pip3] torch==1.5.0
[pip3] torchvision==0.6.0.dev20200326+cu101
[conda] Could not collect


## Additional context



cc @mruberry @VitalyFedyunin