Repository: pytorch
Issue: IsType<T>() ASSERT FAILED [Detectron e2e_mask_rcnn_R-50-C4_1x.yaml]
Labels: caffe2,module: assert failure

## 🐛 Bug

Assertion error when running inference with model successfully trained using the e2e_mask_rcnn_R-50-C4_1x.yaml Detectron config. No such error encountered during training.

## To Reproduce

Steps to reproduce the behavior:
In the Detectron dir:
1. python tools/test_net.py \
--cfg configs/12_2017_baselines/e2e_mask_rcnn_R-50-C4_1x.yaml \
TEST.WEIGHTS tmp/model_final.pkl \
NUM_GPUS 1 \
OUTPUT_DIR tmp/test

After successfully training via:
python tools/train_net.py \ --cfg configs/12_2017_baselines/e2e_mask_rcnn_R-50-C4_1x.yaml \ OUTPUT_DIR tmp/

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->
Traceback (most recent call last):
  File "tools/test_net.py", line 116, in <module>
    check_expected_results=True,
  File "/vol/bitbucket2/rm2815/Detectron/detectron/core/test_engine.py", line 128, in run_inference
    all_results = result_getter()
  File "/vol/bitbucket2/rm2815/Detectron/detectron/core/test_engine.py", line 108, in result_getter
    multi_gpu=multi_gpu_testing
  File "/vol/bitbucket2/rm2815/Detectron/detectron/core/test_engine.py", line 159, in test_net_on_dataset
    weights_file, dataset_name, proposal_file, output_dir, gpu_id=gpu_id
  File "/vol/bitbucket2/rm2815/Detectron/detectron/core/test_engine.py", line 258, in test_net
    model, im, box_proposals, timers
  File "/vol/bitbucket2/rm2815/Detectron/detectron/core/test.py", line 66, in im_detect_all
    model, im, cfg.TEST.SCALE, cfg.TEST.MAX_SIZE, boxes=box_proposals
  File "/vol/bitbucket2/rm2815/Detectron/detectron/core/test.py", line 158, in im_detect_bbox
    workspace.RunNet(model.net.Proto().name)
  File "/vol/bitbucket/rm2815/anaconda3/lib/python3.6/site-packages/caffe2/python/workspace.py", line 237, in RunNet
    StringifyNetName(name), num_iter, allow_fail,
  File "/vol/bitbucket/rm2815/anaconda3/lib/python3.6/site-packages/caffe2/python/workspace.py", line 198, in CallWithExceptionIntercept
    return func(*args, **kwargs)
RuntimeError: IsType<T>() ASSERT FAILED at /opt/conda/conda-bld/pytorch-nightly_1551157756140/work/aten/src/ATen/core/blob.h:77, please report a bug to PyTorch. wrong type for the Blob instance. Blob contains nullptr (uninitialized) while caller expects caffe2::Tensor.
Offending Blob name: gpu_0/conv_rpn_w.
Error from operator: 
input: "gpu_0/res4_5_sum" input: "gpu_0/conv_rpn_w" input: "gpu_0/conv_rpn_b" output: "gpu_0/conv_rpn" name: "" type: "Conv" arg { name: "kernel" i: 3 } arg { name: "order" s: "NCHW" } arg { name: "pad" i: 1 } arg { name: "stride" i: 1 } arg { name: "exhaustive_search" i: 0 } device_option { device_type: 1 device_id: 0 } engine: "CUDNN" (Get at /opt/conda/conda-bld/pytorch-nightly_1551157756140/work/aten/src/ATen/core/blob.h:77)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x45 (0x7f68638f59d5 in /vol/bitbucket/rm2815/anaconda3/lib/python3.6/site-packages/caffe2/python/../../torch/lib/libc10.so)
frame #1: caffe2::Tensor const& caffe2::Blob::Get<caffe2::Tensor>() const + 0xf0 (0x7f686400f750 in /vol/bitbucket/rm2815/anaconda3/lib/python3.6/site-packages/caffe2/python/caffe2_pybind11_state_gpu.cpython-36m-x86_64-linux-gnu.so)
frame #2: caffe2::Tensor const& caffe2::OperatorBase::Input<caffe2::Tensor>(int, c10::DeviceType) + 0x301 (0x7f686408bdf1 in /vol/bitbucket/rm2815/anaconda3/lib/python3.6/site-packages/caffe2/python/caffe2_pybind11_state_gpu.cpython-36m-x86_64-linux-gnu.so)
frame #3: bool caffe2::CudnnConvOp::DoRunWithType<float, float, float, float>() + 0x38 (0x7f682545f428 in /vol/bitbucket/rm2815/anaconda3/lib/python3.6/site-packages/caffe2/python/../../torch/lib/libcaffe2_gpu.so)
frame #4: caffe2::CudnnConvOp::RunOnDevice() + 0x198 (0x7f682544dd08 in /vol/bitbucket/rm2815/anaconda3/lib/python3.6/site-packages/caffe2/python/../../torch/lib/libcaffe2_gpu.so)
frame #5: <unknown function> + 0x13970c5 (0x7f68253ba0c5 in /vol/bitbucket/rm2815/anaconda3/lib/python3.6/site-packages/caffe2/python/../../torch/lib/libcaffe2_gpu.so)
frame #6: caffe2::AsyncNetBase::run(int, int) + 0x144 (0x7f684c460964 in /vol/bitbucket/rm2815/anaconda3/lib/python3.6/site-packages/caffe2/python/../../torch/lib/libcaffe2.so)
frame #7: <unknown function> + 0x16b5549 (0x7f684c467549 in /vol/bitbucket/rm2815/anaconda3/lib/python3.6/site-packages/caffe2/python/../../torch/lib/libcaffe2.so)
frame #8: c10::ThreadPool::main_loop(unsigned long) + 0x273 (0x7f684b47b773 in /vol/bitbucket/rm2815/anaconda3/lib/python3.6/site-packages/caffe2/python/../../torch/lib/libcaffe2.so)
frame #9: <unknown function> + 0xafc5c (0x7f6869201c5c in /vol/bitbucket/rm2815/anaconda3/bin/../lib/libstdc++.so.6)
frame #10: <unknown function> + 0x76db (0x7f6877b6d6db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #11: clone + 0x3f (0x7f687789688f in /lib/x86_64-linux-gnu/libc.so.6)

## Expected behavior
Successful inference using weights from training.
<!-- A clear and concise description of what you expected to happen. -->

## Environment

Collecting environment information...
PyTorch version: 1.1.0
Is debug build: No
CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 18.04.2 LTS
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04) 7.4.0
CMake version: version 3.10.2

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.1
GPU models and configuration: 
GPU 0: TITAN Xp
GPU 1: TITAN Xp
GPU 2: TITAN Xp
GPU 3: TITAN Xp
GPU 4: TITAN Xp
GPU 5: TITAN Xp
GPU 6: TITAN Xp
GPU 7: TITAN Xp
GPU 8: TITAN Xp
GPU 9: TITAN Xp

Nvidia driver version: 390.116
cuDNN version: Could not collect

Versions of relevant libraries:
[pip] numpy==1.16.3
[pip] torch==1.2.0.dev20190601
[pip] torchvision==0.3.0
[conda] blas                      1.0                         mkl  
[conda] cuda80                    1.0                  h205658b_0    pytorch
[conda] mkl                       2019.3                      199  
[conda] mkl_fft                   1.0.12           py37ha843d7b_0  
[conda] mkl_random                1.0.2            py37hd81dba3_0  
[conda] pytorch                   1.1.0           py3.7_cuda9.0.176_cudnn7.5.1_0    pytorch
[conda] pytorch-nightly           1.2.0.dev20190601 py3.7_cuda9.0.176_cudnn7.5.1_0    pytorch
[conda] torchvision               0.3.0           py37_cu9.0.176_1    pytorch

## Additional context
Training works. Inference with below for the tutorial Detectron model works too:
python tools/test_net.py \
--cfg configs/getting_started/tutorial_1gpu_e2e_faster_rcnn_R-50-FPN.yaml \
TEST.WEIGHTS /tmp/detectron-output/train/coco_2014_train/generalized_rcnn/model_final.pkl \
NUM_GPUS 1

<!-- Add any other context about the problem here. -->