Repository: pytorch
Issue: ASSERT FAILED at /opt/conda/conda-bld/pytorch-nightly_1539602533843/work/aten/src/ATen/core/blob.h:79
Labels: caffe2

## 🐛 Bug
Hello Great programmers:
        When I was using FAIR's platform Detectron to do training with *e2e_mask_rcnn_R-101-FPN_3x_gn.yaml* config file, I faced this issue which indicated me to report one BUG to Pytorch.

<!-- A clear and concise description of what the bug is. -->
```
[I net_dag_utils.cc:102] Operator graph pruning prior to chain compute took: 6.1526e-05 secs
[I net_dag_utils.cc:102] Operator graph pruning prior to chain compute took: 5.3939e-05 secs
[I net_dag_utils.cc:102] Operator graph pruning prior to chain compute took: 7.264e-06 secs
[I net_async_base.h:198] Using specified CPU pool size: 4; NUMA node id: -1
[I net_async_base.h:203] Created new CPU pool, size: 4; NUMA node id: -1
[E net_async_base.cc:422] IsType<T>() ASSERT FAILED at /opt/conda/conda-bld/pytorch-nightly_1539602533843/work/aten/src/ATen/core/blob.h:79, please report a bug to PyTorch. wrong type for the Blob instance. Blob contains nullptr (uninitialized) while caller expects caffe2::Tensor.
Offending Blob name: gpu_0/conv1_gn_s.
Error from operator: 
input: "gpu_0/conv1" input: "gpu_0/conv1_gn_s" input: "gpu_0/conv1_gn_b" output: "gpu_0/conv1_gn" output: "gpu_0/conv1_gn_mean" output: "gpu_0/conv1_gn_std" name: "" type: "GroupNorm" arg { name: "use_cudnn" i: 1 } arg { name: "cudnn_exhaustive_search" i: 0 } arg { name: "group" i: 32 } arg { name: "epsilon" f: 1e-05 } device_option { device_type: 1 device_id: 0 } (Get at /opt/conda/conda-bld/pytorch-nightly_1539602533843/work/aten/src/ATen/core/blob.h:79)
frame #0: <unknown function> + 0x277d775 (0x7fa4c1984775 in /home/gengfeng/anaconda3/lib/python3.6/site-packages/caffe2/python/../../torch/lib/libcaffe2_gpu.so)
frame #1: <unknown function> + 0x1321685 (0x7fa4c0528685 in /home/gengfeng/anaconda3/lib/python3.6/site-packages/caffe2/python/../../torch/lib/libcaffe2_gpu.so)
frame #2: caffe2::AsyncNetBase::run(int, int) + 0x16e (0x7fa4ec15f4ee in /home/gengfeng/anaconda3/lib/python3.6/site-packages/caffe2/python/../../torch/lib/libcaffe2.so)
frame #3: <unknown function> + 0x1259972 (0x7fa4ec16e972 in /home/gengfeng/anaconda3/lib/python3.6/site-packages/caffe2/python/../../torch/lib/libcaffe2.so)
frame #4: <unknown function> + 0x124d1cb (0x7fa4ec1621cb in /home/gengfeng/anaconda3/lib/python3.6/site-packages/caffe2/python/../../torch/lib/libcaffe2.so)
frame #5: <unknown function> + 0xafc5c (0x7fa4f6227c5c in /home/gengfeng/anaconda3/bin/../lib/libstdc++.so.6)
frame #6: <unknown function> + 0x76db (0x7fa4fd2806db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #7: clone + 0x3f (0x7fa4fcfa988f in /lib/x86_64-linux-gnu/libc.so.6)
,  op GroupNorm
WARNING workspace.py: 187: Original python traceback for operator `1` in network `generalized_rcnn` in exception above (most recent call last):
WARNING workspace.py: 192:   File "tools/train_net.py", line 132, in <module>
WARNING workspace.py: 192:   File "tools/train_net.py", line 117, in main
WARNING workspace.py: 192:   File "tools/train_net.py", line 127, in test_model
WARNING workspace.py: 192:   File "/home/gengfeng/Desktop/projects/DETECTRON/detectron/core/test_engine.py", line 128, in run_inference
WARNING workspace.py: 192:   File "/home/gengfeng/Desktop/projects/DETECTRON/detectron/core/test_engine.py", line 108, in result_getter
WARNING workspace.py: 192:   File "/home/gengfeng/Desktop/projects/DETECTRON/detectron/core/test_engine.py", line 159, in test_net_on_dataset
WARNING workspace.py: 192:   File "/home/gengfeng/Desktop/projects/DETECTRON/detectron/core/test_engine.py", line 235, in test_net
WARNING workspace.py: 192:   File "/home/gengfeng/Desktop/projects/DETECTRON/detectron/core/test_engine.py", line 328, in initialize_model_from_cfg
WARNING workspace.py: 192:   File "/home/gengfeng/Desktop/projects/DETECTRON/detectron/modeling/model_builder.py", line 124, in create
WARNING workspace.py: 192:   File "/home/gengfeng/Desktop/projects/DETECTRON/detectron/modeling/model_builder.py", line 89, in generalized_rcnn
WARNING workspace.py: 192:   File "/home/gengfeng/Desktop/projects/DETECTRON/detectron/modeling/model_builder.py", line 229, in build_generic_detection_model
WARNING workspace.py: 192:   File "/home/gengfeng/Desktop/projects/DETECTRON/detectron/modeling/optimizer.py", line 54, in build_data_parallel_model
WARNING workspace.py: 192:   File "/home/gengfeng/Desktop/projects/DETECTRON/detectron/modeling/model_builder.py", line 169, in _single_gpu_build_func
WARNING workspace.py: 192:   File "/home/gengfeng/Desktop/projects/DETECTRON/detectron/modeling/FPN.py", line 63, in add_fpn_ResNet101_conv5_body
WARNING workspace.py: 192:   File "/home/gengfeng/Desktop/projects/DETECTRON/detectron/modeling/FPN.py", line 104, in add_fpn_onto_conv_body
WARNING workspace.py: 192:   File "/home/gengfeng/Desktop/projects/DETECTRON/detectron/modeling/ResNet.py", line 48, in add_ResNet101_conv5_body
WARNING workspace.py: 192:   File "/home/gengfeng/Desktop/projects/DETECTRON/detectron/modeling/ResNet.py", line 99, in add_ResNet_convX_body
WARNING workspace.py: 192:   File "/home/gengfeng/Desktop/projects/DETECTRON/detectron/modeling/ResNet.py", line 264, in basic_gn_stem
WARNING workspace.py: 192:   File "/home/gengfeng/Desktop/projects/DETECTRON/detectron/modeling/detector.py", line 450, in ConvGN
WARNING workspace.py: 192:   File "/home/gengfeng/anaconda3/lib/python3.6/site-packages/caffe2/python/cnn.py", line 165, in SpatialGN
WARNING workspace.py: 192:   File "/home/gengfeng/anaconda3/lib/python3.6/site-packages/caffe2/python/brew.py", line 107, in scope_wrapper
WARNING workspace.py: 192:   File "/home/gengfeng/anaconda3/lib/python3.6/site-packages/caffe2/python/helpers/normalization.py", line 206, in spatial_gn
Traceback (most recent call last):
  File "tools/train_net.py", line 132, in <module>
    main()
  File "tools/train_net.py", line 117, in main
    test_model(checkpoints['final'], args.multi_gpu_testing, args.opts)
  File "tools/train_net.py", line 127, in test_model
    check_expected_results=True,
  File "/home/gengfeng/Desktop/projects/DETECTRON/detectron/core/test_engine.py", line 128, in run_inference
    all_results = result_getter()
  File "/home/gengfeng/Desktop/projects/DETECTRON/detectron/core/test_engine.py", line 108, in result_getter
    multi_gpu=multi_gpu_testing
  File "/home/gengfeng/Desktop/projects/DETECTRON/detectron/core/test_engine.py", line 159, in test_net_on_dataset
    weights_file, dataset_name, proposal_file, output_dir, gpu_id=gpu_id
  File "/home/gengfeng/Desktop/projects/DETECTRON/detectron/core/test_engine.py", line 258, in test_net
    model, im, box_proposals, timers
  File "/home/gengfeng/Desktop/projects/DETECTRON/detectron/core/test.py", line 66, in im_detect_all
    model, im, cfg.TEST.SCALE, cfg.TEST.MAX_SIZE, boxes=box_proposals
  File "/home/gengfeng/Desktop/projects/DETECTRON/detectron/core/test.py", line 158, in im_detect_bbox
    workspace.RunNet(model.net.Proto().name)
  File "/home/gengfeng/anaconda3/lib/python3.6/site-packages/caffe2/python/workspace.py", line 219, in RunNet
    StringifyNetName(name), num_iter, allow_fail,
  File "/home/gengfeng/anaconda3/lib/python3.6/site-packages/caffe2/python/workspace.py", line 180, in CallWithExceptionIntercept
    return func(*args, **kwargs)
RuntimeError: IsType<T>() ASSERT FAILED at /opt/conda/conda-bld/pytorch-nightly_1539602533843/work/aten/src/ATen/core/blob.h:79, please report a bug to PyTorch. wrong type for the Blob instance. Blob contains nullptr (uninitialized) while caller expects caffe2::Tensor.
Offending Blob name: gpu_0/conv1_gn_s.
Error from operator: 
input: "gpu_0/conv1" input: "gpu_0/conv1_gn_s" input: "gpu_0/conv1_gn_b" output: "gpu_0/conv1_gn" output: "gpu_0/conv1_gn_mean" output: "gpu_0/conv1_gn_std" name: "" type: "GroupNorm" arg { name: "use_cudnn" i: 1 } arg { name: "cudnn_exhaustive_search" i: 0 } arg { name: "group" i: 32 } arg { name: "epsilon" f: 1e-05 } device_option { device_type: 1 device_id: 0 } (Get at /opt/conda/conda-bld/pytorch-nightly_1539602533843/work/aten/src/ATen/core/blob.h:79)
frame #0: <unknown function> + 0x277d775 (0x7fa4c1984775 in /home/gengfeng/anaconda3/lib/python3.6/site-packages/caffe2/python/../../torch/lib/libcaffe2_gpu.so)
frame #1: <unknown function> + 0x1321685 (0x7fa4c0528685 in /home/gengfeng/anaconda3/lib/python3.6/site-packages/caffe2/python/../../torch/lib/libcaffe2_gpu.so)
frame #2: caffe2::AsyncNetBase::run(int, int) + 0x16e (0x7fa4ec15f4ee in /home/gengfeng/anaconda3/lib/python3.6/site-packages/caffe2/python/../../torch/lib/libcaffe2.so)
frame #3: <unknown function> + 0x1259972 (0x7fa4ec16e972 in /home/gengfeng/anaconda3/lib/python3.6/site-packages/caffe2/python/../../torch/lib/libcaffe2.so)
frame #4: <unknown function> + 0x124d1cb (0x7fa4ec1621cb in /home/gengfeng/anaconda3/lib/python3.6/site-packages/caffe2/python/../../torch/lib/libcaffe2.so)
frame #5: <unknown function> + 0xafc5c (0x7fa4f6227c5c in /home/gengfeng/anaconda3/bin/../lib/libstdc++.so.6)
frame #6: <unknown function> + 0x76db (0x7fa4fd2806db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #7: clone + 0x3f (0x7fa4fcfa988f in /lib/x86_64-linux-gnu/libc.so.6)
```
## To Reproduce

Steps to reproduce the behavior:

1.
1.
1.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior
no exception or bug for training and testing.
<!-- A clear and concise description of what you expected to happen. -->

## Environment

Please copy and paste the output from our
[environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py)
(or fill out the checklist below manually).

You can get the script and run it with:
```
wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
```
gengfeng@ai-work-4:~/Downloads$ python collect_env.py 
Collecting environment information...
PyTorch version: 1.0.0.dev20181015
Is debug build: No
CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 18.04.1 LTS
GCC version: (Ubuntu 5.5.0-12ubuntu1) 5.5.0 20171010
CMake version: version 3.11.4

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 9.0.176
GPU models and configuration: GPU 0: GeForce GTX 1080
Nvidia driver version: 390.87
cuDNN version: Probably one of the following:
/usr/local/cuda-9.0/lib64/libcudnn.so
/usr/local/cuda-9.0/lib64/libcudnn.so.7
/usr/local/cuda-9.0/lib64/libcudnn.so.7.2.1
/usr/local/cuda-9.0/lib64/libcudnn_static.a
/usr/local/cuda-9.1/lib64/libcudnn.so
/usr/local/cuda-9.1/lib64/libcudnn.so.7.1.3
/usr/local/cuda-9.1/lib64/libcudnn_static.a

Versions of relevant libraries:
[pip] Could not collect
[conda] cuda91                    1.0                  h4c16780_0    pytorch
[conda] pytorch-nightly           1.0.0.dev20181015 py3.6_cuda9.0.176_cudnn7.1.2_0    pytorch
[conda] torch                     0.4.0                     <pip>
[conda] torchvision               0.2.1                     <pip>
```

My own configs *e2e_mask_rcnn_R-101-FPN_3x_gn.yaml*
MODEL:
  TYPE: generalized_rcnn
  CONV_BODY: FPN.add_fpn_ResNet101_conv5_body
  NUM_CLASSES: 2
  FASTER_RCNN: True
  MASK_ON: True
NUM_GPUS: 1
SOLVER:
  WEIGHT_DECAY: 0.0001
  LR_POLICY: steps_with_decay
  BASE_LR: 0.002
  GAMMA: 0.1
  MAX_ITER: 70000
  STEPS: [0, 40000, 60000]
FPN:
  FPN_ON: True
  MULTILEVEL_ROIS: True
  MULTILEVEL_RPN: True
  USE_GN: True  # Note: use GN on the FPN-specific layers
RESNETS:
  STRIDE_1X1: False  # default True for MSRA; False for C2 or Torch models
  TRANS_FUNC: bottleneck_gn_transformation  # Note: this is a GN bottleneck transform
  STEM_FUNC: basic_gn_stem  # Note: this is a GN stem
  SHORTCUT_FUNC: basic_gn_shortcut  # Note: this is a GN shortcut
FAST_RCNN:
  ROI_BOX_HEAD: fast_rcnn_heads.add_roi_Xconv1fc_gn_head  # Note: this is a Conv GN head
  ROI_XFORM_METHOD: RoIAlign
  ROI_XFORM_RESOLUTION: 7
  ROI_XFORM_SAMPLING_RATIO: 2
MRCNN:
  ROI_MASK_HEAD: mask_rcnn_heads.mask_rcnn_fcn_head_v1up4convs_gn  # Note: this is a GN mask head
  RESOLUTION: 28  # (output mask resolution) default 14
  ROI_XFORM_METHOD: RoIAlign
  ROI_XFORM_RESOLUTION: 14  # default 7
  ROI_XFORM_SAMPLING_RATIO: 2  # default 0
  DILATION: 1  # default 2
  CONV_INIT: MSRAFill  # default GaussianFill
TRAIN:
  WEIGHTS: https://s3-us-west-2.amazonaws.com/detectron/ImageNetPretrained/47592356/R-101-GN.pkl  # Note: a GN pre-trained model
  DATASETS: ('coco_labelme_train',)
  SCALES: (700,)
  MAX_SIZE: 1333
  BATCH_SIZE_PER_IM: 64
  RPN_PRE_NMS_TOP_N: 2000  # Per FPN level
TEST:
  DATASETS: ('coco_labelme_val',)
  SCALE: 700
  MAX_SIZE: 1333
  NMS: 0.5
  RPN_PRE_NMS_TOP_N: 1000  # Per FPN level
  RPN_POST_NMS_TOP_N: 1000
OUTPUT_DIR: .
```
## Additional context

<!-- Add any other context about the problem here. -->
By the way, *e2e_mask_rcnn_R-50-FPN_1x.yaml* config works fine for me.

Waiting your response, thank you .