Repository: pytorch
Issue: When running GPT trainning with megatron,  the program quit due to torch.distributed.elastic.agent.server.api:Received 1 death signal, shutting down workers
Labels: oncall: distributed,module: elastic

### 🐛 Describe the bug

1.When running GPT trainning with megatron, the program quit due to torch.distributed.elastic.agent.server.api:Received 1 death signal, shutting down workers
2.code
Megatron-LM github branch master and I changed /Megatron-LM/megatron/tokenizer/bert_tokenization.py and /Megatron-LM/megatron/tokenizer/tokenizer.py for berttokenizer data preprocess needs.
[tokenizer.zip](https://github.com/NVIDIA/Megatron-LM/files/9035644/tokenizer.zip)
[bert_tokenization.zip](https://github.com/NVIDIA/Megatron-LM/files/9035645/bert_tokenization.zip)

3.training data ~103MB
[vocab_processed.txt](https://github.com/NVIDIA/Megatron-LM/files/9035635/vocab_processed.txt)
[my-gpt2_test_0704_text_document.zip](https://github.com/NVIDIA/Megatron-LM/files/9035638/my-gpt2_test_0704_text_document.zip)
my-gpt2_test_0704_text_document.bin is ~103MB which exceed size limit, if you need , i can send it.

4.bash 0704_gpt_train.sh
[0704_gpt_train.zip](https://github.com/NVIDIA/Megatron-LM/files/9035651/0704_gpt_train.zip)

5.env:
linux:Linux version 4.15.0-167-generic (buildd@lcy02-amd64-045) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04))
[python env.txt](https://github.com/pytorch/pytorch/files/9035632/python.env.txt)

6.error log:
[0704.log](https://github.com/pytorch/pytorch/files/9035617/0704.log)

### Versions

Collecting environment information...
PyTorch version: 1.11.0+cu102
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.27

Python version: 3.8.13 (default, Mar 28 2022, 11:38:47)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-4.15.0-167-generic-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: 10.2.89
GPU models and configuration: 
GPU 0: Tesla V100S-PCIE-32GB
GPU 1: Tesla V100S-PCIE-32GB
GPU 2: Tesla V100S-PCIE-32GB
GPU 3: Tesla V100S-PCIE-32GB
GPU 4: Tesla V100S-PCIE-32GB
GPU 5: Tesla V100S-PCIE-32GB
GPU 6: Tesla V100S-PCIE-32GB
GPU 7: Tesla V100S-PCIE-32GB

Nvidia driver version: 470.103.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.2.4
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.2.4
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.2.4
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.2.4
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.2.4
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.2.4
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.2.4
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.22.3
[pip3] torch==1.11.0
[pip3] torchaudio==0.11.0
[pip3] torchvision==0.12.0
[conda] blas                      1.0                         mkl    defaults
[conda] cudatoolkit               10.2.89             h713d32c_10    conda-forge
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.4.0           h06a4308_640    defaults
[conda] mkl-service               2.4.0            py38h95df7f1_0    conda-forge
[conda] mkl_fft                   1.3.1            py38h8666266_1    conda-forge
[conda] mkl_random                1.2.2            py38h1abd341_0    conda-forge
[conda] numpy                     1.22.3           py38he7a7128_0    defaults
[conda] numpy-base                1.22.3           py38hf524024_0    defaults
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torch                     1.11.0                   pypi_0    pypi
[conda] torchaudio                0.11.0               py38_cu102    pytorch
[conda] torchvision               0.12.0               py38_cu102    pytorch

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501