Repository: pytorch
Issue: Fixing pytorch distributed training (DDP) code send a SIGKILL signal on its own on most version of pytorch 1.6.0, 1.7.0, 1.7.1?
Labels: oncall: distributed,triaged

## 🐛 Bug

In an attempt to fix error (https://github.com/pytorch/pytorch/issues/54550) I decided to downgrade to multiple pytorch and cuda version (mainly 1.6.0, 1.7.0, 1.7.1 and using cuda 10.2, 11.0 & 11.1). I will only paste the info for 1 case as you will find at the end of the issue since for those version the error is exactly the same (random SIGKILL without explanation). 

```
('lr_reduce_steps', 3)
('validation', True)
('rank', -1)
('master_port', '46209')
('device', device(type='cuda'))
('jobid', '-1_pid_22342')
('slurm_jobid', -1)
('slurm_array_task_id', -1)
('condor_jobid', -1)
('current_time', 'Mar26_19-44-40')
('tb_dir', PosixPath('/home/miranda9/data/logs/logs_Mar26_19-44-40_jobid_-1_pid_22342/tb'))
('gpu_name', 'GeForce GTX TITAN X')
('PID', '22342')
torch.cuda.device_count()=2

opts.world_size=2

ABOUT TO SPAWN WORKERS
done setting sharing strategy...next mp.spawn
rank=0
mp.current_process()=<SpawnProcess name='SpawnProcess-1' parent=22342 started>
os.getpid()=22426
setting up rank=0 (with world_size=2)
MASTER_ADDR='127.0.0.1'
46209
backend='nccl'
rank=1
mp.current_process()=<SpawnProcess name='SpawnProcess-2' parent=22342 started>
os.getpid()=22427
setting up rank=1 (with world_size=2)
MASTER_ADDR='127.0.0.1'
46209
backend='nccl'
--> done setting up rank=1
setup process done for rank=1
--> done setting up rank=0
setup process done for rank=0
2021-03-26 19:44:46.630599: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
DEBUG:tensorflow:Falling back to TensorFlow client; we recommended you install the Cloud TPU client directly with pip install cloud-tpu-client.
Starting training...

n_epoch=0
i=0: loss=tensor(11.1175, device='cuda:0', grad_fn=<NllLossBackward>)
i=1: loss=tensor(10.8209, device='cuda:0', grad_fn=<NllLossBackward>)
i=2: loss=tensor(10.2632, device='cuda:0', grad_fn=<NllLossBackward>)
i=3: loss=tensor(10.0895, device='cuda:0', grad_fn=<NllLossBackward>)
...
i=175: loss=tensor(7.8008, device='cuda:0', grad_fn=<NllLossBackward>)
i=176: loss=tensor(7.6099, device='cuda:0', grad_fn=<NllLossBackward>)
i=177: loss=tensor(6.5869, device='cuda:0', grad_fn=<NllLossBackward>)
i=178: loss=tensor(7.2652, device='cuda:0', grad_fn=<NllLossBackward>)
i=179: loss=tensor(6.9902, device='cuda:0', grad_fn=<NllLossBackward>)
i=180: loss=tensor(7.1965, device='cuda:0', grad_fn=<NllLossBackward>)
Traceback (most recent call last):
  File "/home/miranda9/ML4Coq/ml4coq-proj/embeddings_zoo/tree_nns/main_brando.py", line 279, in <module>
    print(f'-----> device = {torch.device("cuda" if torch.cuda.is_available() else "cpu")}\n')
  File "/home/miranda9/ML4Coq/ml4coq-proj/embeddings_zoo/tree_nns/main_brando.py", line 188, in main_distributed
    print('done setting sharing strategy...next mp.spawn')
  File "/home/miranda9/miniconda3/envs/metalearningpy1.6/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 200, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/home/miranda9/miniconda3/envs/metalearningpy1.6/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 158, in start_processes
    while not context.join():
  File "/home/miranda9/miniconda3/envs/metalearningpy1.6/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 106, in join
    raise Exception(
Exception: process 0 terminated with signal SIGKILL
```

## To Reproduce

Train DDP with any model for a long time. Don't save info or do anything else that is fancy, just train with DDP.

## Expected behavior

Train and converge i 1 day more or less as it used to do.

## Environment

```
(metalearningpy1.6) miranda9~/ML4Coq $ python collect_env.py
Collecting environment information...
PyTorch version: 1.6.0
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: CentOS Linux 7 (Core) (x86_64)
GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39)
Clang version: 3.4.2 (tags/RELEASE_34/dot2-final)
CMake version: version 2.8.12.2

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 11.1.105
GPU models and configuration: 
GPU 0: GeForce GTX TITAN X
GPU 1: GeForce GTX TITAN X
GPU 2: GeForce GTX TITAN X
GPU 3: GeForce GTX TITAN X
GPU 4: GeForce GTX TITAN X
GPU 5: GeForce GTX TITAN X
GPU 6: GeForce GTX TITAN X
GPU 7: GeForce GTX TITAN X

Nvidia driver version: 450.36.06
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] torch==1.6.0
[pip3] torchmeta==1.7.0
[pip3] torchtext==0.8.0a0+c851c3e
[pip3] torchvision==0.7.0
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               10.2.89              hfd86e86_1  
[conda] mkl                       2020.2                      256  
[conda] mkl-service               2.3.0            py38he904b0f_0  
[conda] mkl_fft                   1.3.0            py38h54f3939_0  
[conda] mkl_random                1.1.1            py38h0573a6f_0  
[conda] numpy                     1.19.2           py38h54aff64_0  
[conda] numpy-base                1.19.2           py38hfa32c7d_0  
[conda] pytorch                   1.6.0           py3.8_cuda10.2.89_cudnn7.6.5_0    pytorch
[conda] torchmeta                 1.7.0                    pypi_0    pypi
[conda] torchtext                 0.7.0                      py38    pytorch
[conda] torchvision               0.7.0                py38_cu102    pytorch
```

## Additional context

this was ran as an attempt to avoid the error that happens for me on the newer version 1.8.0 and related issues here:

- https://github.com/pytorch/pytorch/issues/54550
- https://stackoverflow.com/questions/66807131/how-to-solve-the-famous-unhandled-cuda-error-nccl-version-2-7-8-error
- https://stackoverflow.com/questions/66820865/why-does-my-pytorch-distributed-training-ddp-code-send-a-sigkill-signal-on-its


cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @agolynski @SciPioneer @H-Huang @mrzzd @cbalioglu