Repository: pytorch
Issue: Caffe2: executor_test.py::ExecutorGPUResNetTest fails with an AssertionError
Labels: caffe2

## 🐛 Bug

The `executor_test.py::ExecutorGPUResNetTest::test_executor` test case fails on both x86 and Power9 with an error about a blob be completely different on different runs.

## To Reproduce

Steps to reproduce the behavior:

1. Install Caffe2
2. Run `python caffe2/python/test/executor_test.py ExecutorGPUResNetTest`

The `test_executor` case fails with an error about `comp_2_sum_3_grad_autosplit_0` differing in parallel runs. The full output is below:

```
$ python executor_test.py ExecutorGPUResNetTest
Ignoring @/caffe2/caffe2/contrib/nccl:nccl_ops as it is not a valid file.
Ignoring @/caffe2/caffe2/contrib/gloo:gloo_ops as it is not a valid file.
Ignoring @/caffe2/caffe2/contrib/gloo:gloo_ops_gpu as it is not a valid file.
INFO:caffe2.python.net_drawer:Cannot import pydot, which is required for drawing a network. This can usually be installed in python with "pip install pydot". Also, pydot requires graphviz to convert dot files to pdf: in ubuntu, this can usually be installed with "sudo apt-get install graphviz".
net_drawer will not run correctly. Please install the correct dependencies.
[E init_intrinsics_check.cc:43] CPU feature avx is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.
[E init_intrinsics_check.cc:43] CPU feature avx2 is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.
[E init_intrinsics_check.cc:43] CPU feature fma is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.
Trying example: test_executor(self=<__main__.ExecutorGPUResNetTest testMethod=test_executor>, executor='parallel', num_workers=8)
INFO:data_parallel_model:Parallelizing model for devices: [0, 1]
INFO:data_parallel_model:Create input and model training operators
INFO:data_parallel_model:Model for GPU : 0
/home/cphofer/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/caffe2/python/brew.py:97: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec()
  var_names, _, varkw, _= inspect.getargspec(func)
INFO:data_parallel_model:Model for GPU : 1
INFO:data_parallel_model:Adding gradient operators
INFO:data_parallel_model:Add gradient all-reduces for SyncSGD
INFO:data_parallel_model:Post-iteration operators for updating params
INFO:data_parallel_model:Calling optimizer builder function
INFO:data_parallel_model:Add initial parameter sync
WARNING:data_parallel_model:------- DEPRECATED API, please use data_parallel_model.OptimizeGradientMemory() ----- 
/home/cphofer/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/caffe2/python/memonger.py:55: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
  log.warn("NOTE: Executing memonger to optimize gradient memory")
WARNING:memonger:NOTE: Executing memonger to optimize gradient memory
[I memonger.cc:236] Remapping 108 using 14 shared blobs.
INFO:memonger:Memonger memory optimization took 0.030774593353271484 secs
WARNING:memonger:NOTE: Executing memonger to optimize gradient memory
[I memonger.cc:236] Remapping 108 using 14 shared blobs.
INFO:memonger:Memonger memory optimization took 0.04712033271789551 secs
Finished iteration 1/1 (1.90 images/sec)
[I net_dag_utils.cc:102] Operator graph pruning prior to chain compute took: 0.00131458 secs
[I net_parallel.cc:87] Initialized parallel net: 'test_resnet50', #ops: 1675, #chains: 1524, #workers: 8, dfs scheduling: 0, task graph engine: futures
[I net_async_base.h:205] Using specified CUDA pool size: 8; device id: 0
[I net_async_base.h:218] Created shared CUDA pool, size: 8; device id: 0
[I net_async_base.h:205] Using specified CPU pool size: 8; device id: -1
[I net_async_base.h:218] Created shared CPU pool, size: 8; device id: -1
[I net_async_base.h:205] Using specified CUDA pool size: 8; device id: 1
[I net_async_base.h:218] Created shared CUDA pool, size: 8; device id: 1
Finished iteration 1/1 (44.82 images/sec)
Traceback (most recent call last):
  File "/home/cphofer/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/hypothesis/core.py", line 672, in evaluate_test_data
    result = self.execute(data, collect=True)
  File "/home/cphofer/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/hypothesis/core.py", line 587, in execute
    result = self.test_runner(data, run)
  File "/home/cphofer/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/hypothesis/executors.py", line 58, in default_new_style_executor
    return function(data)
  File "/home/cphofer/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/hypothesis/core.py", line 583, in run
    return test(*args, **kwargs)
  File "executor_test.py", line 53, in test_executor
    num_workers=st.sampled_from([8]))
  File "/home/cphofer/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/hypothesis/core.py", line 525, in test
    result = self.test(*args, **kwargs)
  File "executor_test.py", line 67, in test_executor
    model_run_func=run_model,
  File "/home/cphofer/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/caffe2/python/test/executor_test_util.py", line 268, in compare_executors
    "Blob {} differs in {} run".format(blob_name, test_executor))
  File "/home/cphofer/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/numpy/testing/nose_tools/utils.py", line 857, in assert_array_equal
    verbose=verbose, header='Arrays are not equal')
  File "/home/cphofer/anaconda3/envs/pytorch_env/lib/python3.6/site-packages/numpy/testing/nose_tools/utils.py", line 781, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not equal
Blob _gpu_0/comp_2_sum_3_grad_autosplit_0 differs in parallel run
(mismatch 100.0%)
 x: array([[[[ 1.008181e-04, -6.411409e-05,  5.045671e-05, ...,
          -6.273524e-05,  1.340966e-04,  6.698444e-05],
         [-1.703523e-04, -2.880513e-04,  3.850183e-05, ...,...
 y: array([[[[-3.788031e-05,  1.195416e-04,  5.127888e-05, ...,
          -1.146530e-04,  1.209393e-04,  3.692786e-05],
         [-6.415213e-05,  2.887475e-05,  1.446379e-04, ...,...
```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior

Test case should pass

## Environment

PyTorch version: 1.1.0
Is debug build: No
CUDA used to build PyTorch: 10.1.168

OS: Red Hat
GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36)
CMake version: version 2.8.12.2

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration: 
GPU 0: Tesla V100-PCIE-16GB
GPU 1: Tesla V100-PCIE-16GB

Nvidia driver version: 418.40.04
cuDNN version: Could not collect

Versions of relevant libraries:
[pip] numpy==1.14.6
[pip] torch==1.1.0
[pip] torchvision==0.2.2
[conda] blas                      1.0                         mkl  
[conda] magma                     2.5.0
[conda] mkl                       2018.0.3
[conda] mkl_fft                   1.0.6
[conda] mkl_random                1.0.1
[conda] pytorch                   1.1.0

### Power

PyTorch version: 1.1.0
Is debug build: No
CUDA used to build PyTorch: 10.1.168

OS: Red Hat Enterprise Linux Server 7.6 (Maipo)
GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36)
CMake version: Could not collect

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 10.1.105
GPU models and configuration: 
GPU 0: Tesla V100-SXM2-16GB
GPU 1: Tesla V100-SXM2-16GB
GPU 2: Tesla V100-SXM2-16GB
GPU 3: Tesla V100-SXM2-16GB

Nvidia driver version: 418.67
cuDNN version: 7.5.1

Versions of relevant libraries:
[pip] numpy==1.14.5
[pip] torch==1.1.0
[pip] torchvision==0.2.2
[conda] magma                     2.5.0
[conda] pytorch                   1.1.0