Repository: pytorch
Issue: Segmentation Fault (core dumped) with 1.4.0
Labels: needs reproduction,triaged

## 🐛 Bug

Running a model training in the rapids docker container with 1.4.0 is throwing segmentation fault error while calling `.backward()` function whereas it is working as expected with 1.3.1. please find the below logs.

## Unexpected behavior
```
(rapids) root@dgx:/rapids/my_data/git_repo/clx# nvidia-smi
Fri Jan 24 08:48:15 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.00    Driver Version: 418.87.00    CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:06:00.0 Off |                    0 |
| N/A   31C    P0    42W / 300W |      0MiB / 32480MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...  On   | 00000000:07:00.0 Off |                    0 |
| N/A   34C    P0    43W / 300W |      0MiB / 32480MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla V100-SXM2...  On   | 00000000:0A:00.0 Off |                    0 |
| N/A   33C    P0    42W / 300W |      0MiB / 32480MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
(rapids) root@dgx:/rapids/my_data/git_repo/clx# conda list | grep torch
torch                     1.4.0                    pypi_0    pypi
(rapids) root@dgx:/rapids/my_data/git_repo/clx# python examples/train_dga_detector.py --help
usage: train_dga_detector.py [-h]
                             train_file_path test_file_path output_dir
                             batch_size epoch

Model training runner

positional arguments:
  train_file_path  Input dataset for model training
  test_file_path   Input dataset for model testing
  output_dir       Output directory to store trained model
  batch_size       Dividing dataset into number of batches or sets or parts
  epoch            One epoch is when an entire dataset is passed forward and
                   backward through the neural network only once

optional arguments:
  -h, --help       show this help message and exit
(rapids) root@dgx:/rapids/my_data/git_repo/clx# python examples/train_dga_detector.py /rapids/my_data/dga/data/train2.csv /rapids/my_data/dga/data/test.csv /rapids/my_data/test_dga/ 10000 3
INFO:train_file_path : /rapids/my_data/dga/data/train2.csv
INFO:test_file_path : /rapids/my_data/dga/data/test.csv
INFO:Verify if output directory `/rapids/my_data/test_dga/` is already exists.
INFO:Output directory `/rapids/my_data/test_dga/` does not exists.
INFO:Creating directory `/rapids/my_data/test_dga/` to store trained models.
INFO:**********
INFO:Epoch: 1
INFO:**********
/opt/conda/envs/rapids/lib/python3.7/site-packages/cudf/io/dlpack.py:82: UserWarning: WARNING: cuDF to_dlpack() produces column-major (Fortran order) output. If the output tensor needs to be row major, transpose the output of this function.
  return cpp_dlpack.to_dlpack(gdf_cols)
[100000/1468974 (7%)]	Loss: 5368.84
[200000/1468974 (14%)]	Loss: 5238.29
[300000/1468974 (20%)]	Loss: 5266.80
[400000/1468974 (27%)]	Loss: 5503.81
[500000/1468974 (34%)]	Loss: 5675.76
[600000/1468974 (41%)]	Loss: 5730.03
[700000/1468974 (48%)]	Loss: 5778.36
[800000/1468974 (54%)]	Loss: 5797.21
[900000/1468974 (61%)]	Loss: 5762.53
[1000000/1468974 (68%)]	Loss: 5736.77
[1100000/1468974 (75%)]	Loss: 5621.45
[1200000/1468974 (82%)]	Loss: 5489.19
Segmentation fault (core dumped)
```
## Expected behavior
```
(rapids) root@dgx:/rapids/my_data/git_repo/clx# pip install torch==1.3.1
Collecting torch==1.3.1
  Downloading https://files.pythonhosted.org/packages/f9/34/2107f342d4493b7107a600ee16005b2870b5a0a5a165bdf5c5e7168a16a6/torch-1.3.1-cp37-cp37m-manylinux1_x86_64.whl (734.6MB)
     |████████████████████████████████| 734.6MB 23kB/s 
Requirement already satisfied: numpy in /opt/conda/envs/rapids/lib/python3.7/site-packages (from torch==1.3.1) (1.17.3)
Installing collected packages: torch
  Found existing installation: torch 1.4.0
    Uninstalling torch-1.4.0:
      Successfully uninstalled torch-1.4.0
Successfully installed torch-1.3.1
(rapids) root@dgx:/rapids/my_data/git_repo/clx# python examples/train_dga_detector.py /rapids/my_data/dga/data/train2.csv /rapids/my_data/dga/data/test.csv /rapids/my_data/test_dga/ 10000 3
INFO:train_file_path : /rapids/my_data/dga/data/train2.csv
INFO:test_file_path : /rapids/my_data/dga/data/test.csv
INFO:Verify if output directory `/rapids/my_data/test_dga/` is already exists.
INFO:**********
INFO:Epoch: 1
INFO:**********
/opt/conda/envs/rapids/lib/python3.7/site-packages/cudf/io/dlpack.py:82: UserWarning: WARNING: cuDF to_dlpack() produces column-major (Fortran order) output. If the output tensor needs to be row major, transpose the output of this function.
  return cpp_dlpack.to_dlpack(gdf_cols)
[100000/1468974 (7%)]	Loss: 4696.43
[200000/1468974 (14%)]	Loss: 5147.29
[300000/1468974 (20%)]	Loss: 4525.80
[400000/1468974 (27%)]	Loss: 4116.06
[500000/1468974 (34%)]	Loss: 3795.70
[600000/1468974 (41%)]	Loss: 3486.26
[700000/1468974 (48%)]	Loss: 3230.03
[800000/1468974 (54%)]	Loss: 2969.96
[900000/1468974 (61%)]	Loss: 2875.38
[1000000/1468974 (68%)]	Loss: 2768.05
[1100000/1468974 (75%)]	Loss: 2662.28
[1200000/1468974 (82%)]	Loss: 2584.43
[1300000/1468974 (88%)]	Loss: 2544.63
[1400000/1468974 (95%)]	Loss: 2469.58
Test set: Accuracy: 224095/367244 (0.6102073825576456)

/opt/conda/envs/rapids/lib/python3.7/site-packages/torch/serialization.py:292: UserWarning: Couldn't retrieve source code for container of type DataParallel. It won't be checked for correctness upon loading.
  "type " + obj.__name__ + ". It won't be checked "
/opt/conda/envs/rapids/lib/python3.7/site-packages/torch/serialization.py:292: UserWarning: Couldn't retrieve source code for container of type RNNClassifier. It won't be checked for correctness upon loading.
  "type " + obj.__name__ + ". It won't be checked "
/opt/conda/envs/rapids/lib/python3.7/site-packages/torch/serialization.py:292: UserWarning: Couldn't retrieve source code for container of type Embedding. It won't be checked for correctness upon loading.
  "type " + obj.__name__ + ". It won't be checked "
/opt/conda/envs/rapids/lib/python3.7/site-packages/torch/serialization.py:292: UserWarning: Couldn't retrieve source code for container of type GRU. It won't be checked for correctness upon loading.
  "type " + obj.__name__ + ". It won't be checked "
/opt/conda/envs/rapids/lib/python3.7/site-packages/torch/serialization.py:292: UserWarning: Couldn't retrieve source code for container of type Linear. It won't be checked for correctness upon loading.
  "type " + obj.__name__ + ". It won't be checked "
INFO:**********
INFO:Epoch: 2
INFO:**********
[100000/1468974 (7%)]	Loss: 4053.78
[200000/1468974 (14%)]	Loss: 3097.61
[300000/1468974 (20%)]	Loss: 2553.45
[400000/1468974 (27%)]	Loss: 2228.93
[500000/1468974 (34%)]	Loss: 1973.65
[600000/1468974 (41%)]	Loss: 1785.76
[700000/1468974 (48%)]	Loss: 1629.33
[800000/1468974 (54%)]	Loss: 1527.37
[900000/1468974 (61%)]	Loss: 1532.07
[1000000/1468974 (68%)]	Loss: 1489.94
[1100000/1468974 (75%)]	Loss: 1473.13
[1200000/1468974 (82%)]	Loss: 1464.33
[1300000/1468974 (88%)]	Loss: 1478.28
[1400000/1468974 (95%)]	Loss: 1464.06
Test set: Accuracy: 278392/367244 (0.7580573133938199)

INFO:**********
INFO:Epoch: 3
INFO:**********
[100000/1468974 (7%)]	Loss: 1578.96
[200000/1468974 (14%)]	Loss: 1876.74
[300000/1468974 (20%)]	Loss: 1545.72
[400000/1468974 (27%)]	Loss: 1357.28
[500000/1468974 (34%)]	Loss: 1219.40
[600000/1468974 (41%)]	Loss: 1118.28
[700000/1468974 (48%)]	Loss: 1040.39
[800000/1468974 (54%)]	Loss: 996.32
[900000/1468974 (61%)]	Loss: 971.37
[1000000/1468974 (68%)]	Loss: 956.18
[1100000/1468974 (75%)]	Loss: 968.09
[1200000/1468974 (82%)]	Loss: 1007.30
[1300000/1468974 (88%)]	Loss: 1039.72
[1400000/1468974 (95%)]	Loss: 1047.50
Test set: Accuracy: 329641/367244 (0.8976075851477492)

INFO:Model with highest accuracy (0.8976075851477492) is stored to location /rapids/my_data/test_dga//rnn_classifier_2020-01-24_08_52_23.pth
INFO:Time taken for training a model : 51.769317388534546
(rapids) root@dgx:/rapids/my_data/git_repo/clx# conda list| grep torch
torch                     1.3.1                    pypi_0    pypi
```
## Environment
```
(rapids) root@dgx03:/rapids/my_data/git_repo# python collect_env.py 
Collecting environment information...
PyTorch version: 1.3.1
Is debug build: No
CUDA used to build PyTorch: 10.1.243

OS: NVIDIA DGX Server
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
CMake version: version 3.14.5

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration: 
GPU 0: Tesla V100-SXM2-32GB
GPU 1: Tesla V100-SXM2-32GB
GPU 2: Tesla V100-SXM2-32GB

Nvidia driver version: 418.87.00
cuDNN version: Could not collect

Versions of relevant libraries:
[pip] numpy==1.17.3
[pip] numpydoc==0.9.2
[pip] torch==1.3.1
[conda] torch                     1.3.1                    pypi_0    pypi
```