Repository: pytorch
Issue: Test failure for depthwise3x3_conv
Labels: caffe2

## Bug

Recently I start to get test failure on `depthwise3x3_conv`.
And this happened on both CPU/GPU build, although this is probably not GPU related.

I haven't get chance to dig deeper yet.

```
      Start 89: depthwise3x3_conv_op_test
89/92 Test #89: depthwise3x3_conv_op_test ...............***Failed    0.24 sec
```

Here's the details:

```
Running main() from ../third_party/googletest/googletest/src/gtest_main.cc
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from DEPTHWISE3x3
[ RUN      ] DEPTHWISE3x3.Conv
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0425 01:23:28.802515 49204 depthwise3x3_conv_op_test.cc:54] running N 1 inputC 2 H 95 W 77 outputC 2 kernelH 3 kernelW 3 strideH 1 strideW 1 padT 2 padL 2 padB 2 padR 2 group 2
W0425 01:23:28.803205 49204 init.h:115] Caffe2 GlobalInit should be run before any other API calls.
../caffe2/share/contrib/depthwise/depthwise3x3_conv_op_test.cc:145: Failure
Value of: relErr <= maxRelErr || (relErr > maxRelErr && absErr <= absErrForRelErrFailure)
  Actual: false
Expected: true
25 20 (rel err 0.2222222238779068) (0 0 0 0) running N 1 inputC 2 H 95 W 77 outputC 2 kernelH 3 kernelW 3 strideH 1 strideW 1 padT 2 padL 2 padB 2 padR 2 group 2
../caffe2/share/contrib/depthwise/depthwise3x3_conv_op_test.cc:145: Failure
Value of: relErr <= maxRelErr || (relErr > maxRelErr && absErr <= absErrForRelErrFailure)
  Actual: false
Expected: true
50 45 (rel err 0.10526315867900848) (0 0 0 1) running N 1 inputC 2 H 95 W 77 outputC 2 kernelH 3 kernelW 3 strideH 1 strideW 1 padT 2 padL 2 padB 2 padR 2 group 2
../caffe2/share/contrib/depthwise/depthwise3x3_conv_op_test.cc:145: Failure
Value of: relErr <= maxRelErr || (relErr > maxRelErr && absErr <= absErrForRelErrFailure)
  Actual: false
Expected: true
75 70 (rel err 0.068965516984462738) (0 0 0 2) running N 1 inputC 2 H 95 W 77 outputC 2 kernelH 3 kernelW 3 strideH 1 strideW 1 padT 2 padL 2 padB 2 padR 2 group 2
../caffe2/share/contrib/depthwise/depthwise3x3_conv_op_test.cc:145: Failure
Value of: relErr <= maxRelErr || (relErr > maxRelErr && absErr <= absErrForRelErrFailure)
  Actual: false
Expected: true
75 70 (rel err 0.068965516984462738) (0 0 0 3) running N 1 inputC 2 H 95 W 77 outputC 2 kernelH 3 kernelW 3 strideH 1 strideW 1 padT 2 padL 2 padB 2 padR 2 group 2
../caffe2/share/contrib/depthwise/depthwise3x3_conv_op_test.cc:145: Failure
Value of: relErr <= maxRelErr || (relErr > maxRelErr && absErr <= absErrForRelErrFailure)
  Actual: false
Expected: true
75 70 (rel err 0.068965516984462738) (0 0 0 4) running N 1 inputC 2 H 95 W 77 outputC 2 kernelH 3 kernelW 3 strideH 1 strideW 1 padT 2 padL 2 padB 2 padR 2 group 2
../caffe2/share/contrib/depthwise/depthwise3x3_conv_op_test.cc:145: Failure
Value of: relErr <= maxRelErr || (relErr > maxRelErr && absErr <= absErrForRelErrFailure)
  Actual: false
Expected: true
75 70 (rel err 0.068965516984462738) (0 0 0 5) running N 1 inputC 2 H 95 W 77 outputC 2 kernelH 3 kernelW 3 strideH 1 strideW 1 padT 2 padL 2 padB 2 padR 2 group 2
../caffe2/share/contrib/depthwise/depthwise3x3_conv_op_test.cc:145: Failure
Value of: relErr <= maxRelErr || (relErr > maxRelErr && absErr <= absErrForRelErrFailure)
  Actual: false
Expected: true
75 70 (rel err 0.068965516984462738) (0 0 0 6) running N 1 inputC 2 H 95 W 77 outputC 2 kernelH 3 kernelW 3 strideH 1 strideW 1 padT 2 padL 2 padB 2 padR 2 group 2
../caffe2/share/contrib/depthwise/depthwise3x3_conv_op_test.cc:145: Failure
Value of: relErr <= maxRelErr || (relErr > maxRelErr && absErr <= absErrForRelErrFailure)
  Actual: false
Expected: true
75 70 (rel err 0.068965516984462738) (0 0 0 7) running N 1 inputC 2 H 95 W 77 outputC 2 kernelH 3 kernelW 3 strideH 1 strideW 1 padT 2 padL 2 padB 2 padR 2 group 2
../caffe2/share/contrib/depthwise/depthwise3x3_conv_op_test.cc:145: Failure
Value of: relErr <= maxRelErr || (relErr > maxRelErr && absErr <= absErrForRelErrFailure)
  Actual: false
Expected: true
75 70 (rel err 0.068965516984462738) (0 0 0 8) running N 1 inputC 2 H 95 W 77 outputC 2 kernelH 3 kernelW 3 strideH 1 strideW 1 padT 2 padL 2 padB 2 padR 2 group 2
../caffe2/share/contrib/depthwise/depthwise3x3_conv_op_test.cc:145: Failure
Value of: relErr <= maxRelErr || (relErr > maxRelErr && absErr <= absErrForRelErrFailure)
  Actual: false
Expected: true
75 70 (rel err 0.068965516984462738) (0 0 0 9) running N 1 inputC 2 H 95 W 77 outputC 2 kernelH 3 kernelW 3 strideH 1 strideW 1 padT 2 padL 2 padB 2 padR 2 group 2
../caffe2/share/contrib/depthwise/depthwise3x3_conv_op_test.cc:145: Failure
Value of: relErr <= maxRelErr || (relErr > maxRelErr && absErr <= absErrForRelErrFailure)
  Actual: false
Expected: true
75 70 (rel err 0.068965516984462738) (0 0 0 10) running N 1 inputC 2 H 95 W 77 outputC 2 kernelH 3 kernelW 3 strideH 1 strideW 1 padT 2 padL 2 padB 2 padR 2 group 2
../caffe2/share/contrib/depthwise/depthwise3x3_conv_op_test.cc:145: Failure
Value of: relErr <= maxRelErr || (relErr > maxRelErr && absErr <= absErrForRelErrFailure)
  Actual: false
Expected: true
75 70 (rel err 0.068965516984462738) (0 0 0 11) running N 1 inputC 2 H 95 W 77 outputC 2 kernelH 3 kernelW 3 strideH 1 strideW 1 padT 2 padL 2 padB 2 padR 2 group 2
../caffe2/share/contrib/depthwise/depthwise3x3_conv_op_test.cc:145: Failure
Value of: relErr <= maxRelErr || (relErr > maxRelErr && absErr <= absErrForRelErrFailure)
  Actual: false
Expected: true
75 70 (rel err 0.068965516984462738) (0 0 0 12) running N 1 inputC 2 H 95 W 77 outputC 2 kernelH 3 kernelW 3 strideH 1 strideW 1 padT 2 padL 2 padB 2 padR 2 group 2
../caffe2/share/contrib/depthwise/depthwise3x3_conv_op_test.cc:145: Failure
Value of: relErr <= maxRelErr || (relErr > maxRelErr && absErr <= absErrForRelErrFailure)
  Actual: false
Expected: true
75 70 (rel err 0.068965516984462738) (0 0 0 13) running N 1 inputC 2 H 95 W 77 outputC 2 kernelH 3 kernelW 3 strideH 1 strideW 1 padT 2 padL 2 padB 2 padR 2 group 2
../caffe2/share/contrib/depthwise/depthwise3x3_conv_op_test.cc:145: Failure
Value of: relErr <= maxRelErr || (relErr > maxRelErr && absErr <= absErrForRelErrFailure)
  Actual: false
Expected: true
75 70 (rel err 0.068965516984462738) (0 0 0 14) running N 1 inputC 2 H 95 W 77 outputC 2 kernelH 3 kernelW 3 strideH 1 strideW 1 padT 2 padL 2 padB 2 padR 2 group 2
../caffe2/share/contrib/depthwise/depthwise3x3_conv_op_test.cc:145: Failure
Value of: relErr <= maxRelErr || (relErr > maxRelErr && absErr <= absErrForRelErrFailure)
  Actual: false
Expected: true
75 70 (rel err 0.068965516984462738) (0 0 0 15) running N 1 inputC 2 H 95 W 77 outputC 2 kernelH 3 kernelW 3 strideH 1 strideW 1 padT 2 padL 2 padB 2 padR 2 group 2
../caffe2/share/contrib/depthwise/depthwise3x3_conv_op_test.cc:145: Failure
Value of: relErr <= maxRelErr || (relErr > maxRelErr && absErr <= absErrForRelErrFailure)
  Actual: false
Expected: true
75 70 (rel err 0.068965516984462738) (0 0 0 16) running N 1 inputC 2 H 95 W 77 outputC 2 kernelH 3 kernelW 3 strideH 1 strideW 1 padT 2 padL 2 padB 2 padR 2 group 2
../caffe2/share/contrib/depthwise/depthwise3x3_conv_op_test.cc:145: Failure
Value of: relErr <= maxRelErr || (relErr > maxRelErr && absErr <= absErrForRelErrFailure)
  Actual: false
Expected: true
75 70 (rel err 0.068965516984462738) (0 0 0 17) running N 1 inputC 2 H 95 W 77 outputC 2 kernelH 3 kernelW 3 strideH 1 strideW 1 padT 2 padL 2 padB 2 padR 2 group 2
../caffe2/share/contrib/depthwise/depthwise3x3_conv_op_test.cc:145: Failure
Value of: relErr <= maxRelErr || (relErr > maxRelErr && absErr <= absErrForRelErrFailure)
```

## Environment

 - PyTorch Version (e.g., 1.0): master
 - OS (e.g., Linux): CentOS-7 and Ubuntu-19.04
 - How you installed PyTorch (`conda`, `pip`, source): source
 - Build command you used (if compiling from source): cmake + ninja + gcc8
 - Python version: Stock 3.6 on both distro
 - CUDA/cuDNN version:  10.1
 - GPU models and configuration: Volta