Repository: pytorch
Issue: Bug about distributed launch
Labels: oncall: distributed

### 🐛 Describe the bug

Traceback (most recent call last):                                                                                                                

  File "/opt/conda/lib/python3.8/runpy.py", line 194, in _run_module_as_main                                                                      

    return _run_code(code, main_globals, None,                                                                                                    

  File "/opt/conda/lib/python3.8/runpy.py", line 87, in _run_code                                                                                 

    exec(code, run_globals)                                                                                                                       

  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/launch.py", line 260, in <module>                                                

    main()                                                                                                                                        

  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/launch.py", line 255, in main                                                    

    raise subprocess.CalledProcessError(returncode=process.returncode,                                                                            

subprocess.CalledProcessError: Command '['/opt/conda/bin/python', '-u', 'run_mae_pretraining.py', '--local_rank=7', '--data_path', '/bigdata/data2

1/imagenet21k_train/', '--mask_ratio', '0.75', '--model', 'pretrain_mae_large_patch16_224', '--batch_size', '128', '--opt', 'adamw', '--opt_betas'

, '0.9', '0.95', '--warmup_epochs', '20', '--epochs', '200', '--output_dir', '/work/MAE-pytorch/output/pretrain_mae_large_patch16_224_21k/']' died

 with <Signals.SIGKILL: 9>. 



def main(args):
    utils.init_distributed_mode(args)

    print(args)

    device = torch.device(args.device)

    # fix the seed for reproducibility
    seed = args.seed + utils.get_rank()
    torch.manual_seed(seed)
    np.random.seed(seed)
    # random.seed(seed)

    cudnn.benchmark = True

    model = get_model(args)
    patch_size = model.encoder.patch_embed.patch_size
    print("Patch size = %s" % str(patch_size))
    args.window_size = (args.input_size // patch_size[0], args.input_size // patch_size[1])
    args.patch_size = patch_size

    # get dataset
    dataset_train = build_pretraining_dataset(args)
    print(dataset_train)
  if True:  # args.distributed:
          num_tasks = utils.get_world_size()
          global_rank = utils.get_rank()
          sampler_rank = global_rank
          num_training_steps_per_epoch = len(dataset_train) // args.batch_size // num_tasks
  
          sampler_train = torch.utils.data.DistributedSampler(
              dataset_train, num_replicas=num_tasks, rank=sampler_rank, shuffle=True
          )
          print("Sampler_train = %s" % str(sampler_train))




### Versions

yTorch version: 1.7.1
Is debug build: False
CUDA used to build PyTorch: 11.0
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.27

Python version: 3.8.5 (default, Sep  4 2020, 07:30:14)  [GCC 7.3.0] (64-bit runtime)
Python platform: Linux-4.15.0-163-generic-x86_64-with-glibc2.10
Is CUDA available: True
CUDA runtime version: 11.0.221
GPU models and configuration: 
GPU 0: Tesla V100-SXM2-32GB
GPU 1: Tesla V100-SXM2-32GB
GPU 2: Tesla V100-SXM2-32GB
GPU 3: Tesla V100-SXM2-32GB
GPU 4: Tesla V100-SXM2-32GB
GPU 5: Tesla V100-SXM2-32GB
GPU 6: Tesla V100-SXM2-32GB
GPU 7: Tesla V100-SXM2-32GB

Nvidia driver version: 440.100
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.0.4
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] mypy==0.910
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.19.2
[pip3] torch==1.7.1
[pip3] torchelastic==0.2.1
[pip3] torchvision==0.8.2
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.0.221             h6bb024c_0  
[conda] mkl                       2020.2                      256  
[conda] mkl-service               2.3.0            py38he904b0f_0  
[conda] mkl_fft                   1.2.0            py38h23d657b_0  
[conda] mkl_random                1.1.1            py38h0573a6f_0  
[conda] mypy                      0.910                    pypi_0    pypi
[conda] mypy-extensions           0.4.3                    pypi_0    pypi
[conda] numpy                     1.19.2           py38h54aff64_0  
[conda] numpy-base                1.19.2           py38hfa32c7d_0  
[conda] pytorch                   1.7.1           py3.8_cuda11.0.221_cudnn8.0.5_0    pytorch
[conda] torchelastic              0.2.1                    pypi_0    pypi
[conda] torchvision               0.8.2                py38_cu110    pytorch


cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang