Repository: pytorch
Issue: caffe2: RuntimeError: [enforce fail at reshape_op.h:110]  with Alexnet onnx test with cuda
Labels: caffe2

## 🐛 Bug
we are seeing an intermittent failure in the reshape_op when trying to run the EXAMPLE: END-TO-END ALEXNET FROM PYTORCH TO CAFFE2.

```
Traceback (most recent call last):
  File "/tmp/test.py", line 35, in <module>
    outputs = rep.run(np.random.randn(10, 3, 227, 227).astype(np.float32))
  File "/opt/DL/pytorch/lib/python2.7/site-packages/caffe2/python/onnx/backend_rep.py", line 57, in run
    self.workspace.RunNet(self.predict_net.name)
  File "/opt/DL/pytorch/lib/python2.7/site-packages/caffe2/python/onnx/workspace.py", line 63, in f
    return getattr(workspace, attr)(*args, **kwargs)
  File "/opt/DL/pytorch/lib/python2.7/site-packages/caffe2/python/workspace.py", line 219, in RunNet
    StringifyNetName(name), num_iter, allow_fail,
  File "/opt/DL/pytorch/lib/python2.7/site-packages/caffe2/python/workspace.py", line 180, in CallWithExceptionIntercept
    return func(*args, **kwargs)
RuntimeError: [enforce fail at reshape_op.h:110] total_size == size. 92160 vs -5680358544067434496. Argument `shape` does not agree with the input data. (92160 != -5680358544067434496)Error from operator:
input: "29" input: "36" output: "37" output: "OC2_DUMMY_1" name: "" type: "Reshape" device_option { device_type: 1 device_id: 0 }
```

## To Reproduce
Steps to reproduce the behavior:
Using the example code and the exported alexnet.onnx file  run the following sample.
```
import onnx
import caffe2.python.onnx.backend as backend
import numpy as np
model = onnx.load("alexnet.onnx")

rep = backend.prepare(model, device="CUDA") # or "CPU"
outputs = rep.run(np.random.randn(10, 3, 224, 224).astype(np.float32))
print(outputs[0])
```
## Expected behavior
example code runs successfully every time.

## Environment

 - PyTorch Version (e.g., 1.0): 1.0 rc1 
 - OS (e.g., Linux):  RHEL 7.5 ppc64le
 - How you installed PyTorch (`conda`, `pip`, source): Build from source
 - Build command you used (if compiling from source): DEBUG=1 USE_OPENCV=1 python setup.py
 - Python version: 2.7 or 3.6
 - CUDA/cuDNN version:  10 7.31 
 - GPU models and configuration: v100
 - Any other relevant information:

problem only occurs when use device = cuda, if device = cpu the issue is not see, seen on ppc64le unknown if other platforms are effected.

## Additional context
Checking that very large incorrect value, we see least significant
32-bits of the total size is correct (i.e. 0x16800, or 92160), but the
high order 32-bits are incorrect (should be 0x0, but contain data):

```
$ echo "obase=16; 2^64 -5680358544067434496" | bc
B12B500000016800

$ echo "ibase=16; 16800" | bc
92160
```

If we build Caffe2 to crash here, we find the cause of the problem seems
to be that the shape information copied back from the GPU is damaged,
for example in one run we see:

```
(gdb) print actual_new_shape                                                                                                                                        
$6 = {<std::_Vector_base<long, std::allocator<long> >> = {                                                                                                          
    _M_impl = {<std::allocator<long>> = {<__gnu_cxx::new_allocator<long>> = {<No data fields>}, <No data fields>}, _M_start = 0x7ffdf836e930,                       
      _M_finish = 0x7ffdf836e940, _M_end_of_storage = 0x7ffdf836e940}}, <No data fields>}

(gdb) x/4x 0x7ffdf836e930                                                                                                                                           
0x7ffdf836e930: 0x0000000a      0x00000005      0x00002400      0x00000000
```

Here, 2 values (0x0a / 10, and 0x2400 / 9216) should have been copied in
from the GPU, but instead the most-significant 32-bits of the "10" value
have been overlayed with "0x05", resulting in a final apparent value of
0x0000 0005 0000 000a (21474836490).

Since the problem is intermittent, it's not clear whether the overwrite
is always happening (but mostly happens to be 0x0) or only happens
sometimes.

<!-- Add any other context about the problem here. -->