Repository: pytorch
Issue: [distibuted] torch.distributed.new_group failed with Segmentation fault
Labels: oncall: distributed

### 🐛 Describe the bug

I meet the segment fault when use Megatron-LM firstly, and then I reproduce it using simple torch code.

```
import torch
import os
os.environ['MASTER_ADDR']='{my_ip}'
os.environ['MASTER_PORT']='6001'
torch.distributed.init_process_group(backend='nccl', world_size=2, rank=0)  # run torch.distributed.init_process_group(backend='nccl', world_size=2, rank=1) with another process
torch.distributed.new_group(backend='mpi')
```

then, get the error

```
[5485-6c694df6db-6hsps:936  :0:936] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x28)
[5485-6c694df6db-6hsps:935  :0:935] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x28)
==== backtrace (tid:    936) ====
 0 0x0000000000043090 killpg()  ???:0
 1 0x0000000000044400 ompi_dpm_group_is_dyn()  /build-result/src/hpcx-v2.13-gcc-inbox-ubuntu20.04-cuda11-gdrcopy2-nccl2.12-x86_64/ompi-5abd86cc8c5d75c5fe7894b379515d97839c1416/ompi/dpm/dpm.c:1267
 2 0x0000000000044400 ompi_dpm_group_is_dyn()  /build-result/src/hpcx-v2.13-gcc-inbox-ubuntu20.04-cuda11-gdrcopy2-nccl2.12-x86_64/ompi-5abd86cc8c5d75c5fe7894b379515d97839c1416/ompi/dpm/dpm.c:1268
 3 0x0000000000044400 ompi_dpm_mark_dyncomm()  /build-result/src/hpcx-v2.13-gcc-inbox-ubuntu20.04-cuda11-gdrcopy2-nccl2.12-x86_64/ompi-5abd86cc8c5d75c5fe7894b379515d97839c1416/ompi/dpm/dpm.c:1299
 4 0x0000000000034295 ompi_comm_set_nb()  /build-result/src/hpcx-v2.13-gcc-inbox-ubuntu20.04-cuda11-gdrcopy2-nccl2.12-x86_64/ompi-5abd86cc8c5d75c5fe7894b379515d97839c1416/ompi/communicator/comm.c:215
 5 0x00000000000345ba ompi_comm_set()  /build-result/src/hpcx-v2.13-gcc-inbox-ubuntu20.04-cuda11-gdrcopy2-nccl2.12-x86_64/ompi-5abd86cc8c5d75c5fe7894b379515d97839c1416/ompi/communicator/comm.c:116
 6 0x0000000000034e53 ompi_comm_create()  /build-result/src/hpcx-v2.13-gcc-inbox-ubuntu20.04-cuda11-gdrcopy2-nccl2.12-x86_64/ompi-5abd86cc8c5d75c5fe7894b379515d97839c1416/ompi/communicator/comm.c:344
 7 0x000000000006e5fa PMPI_Comm_create()  /build-result/src/hpcx-v2.13-gcc-inbox-ubuntu20.04-cuda11-gdrcopy2-nccl2.12-x86_64/ompi-5abd86cc8c5d75c5fe7894b379515d97839c1416/ompi/mpi/c/profile/pcomm_create.c:66
 8 0x0000000005029ccd c10d::ProcessGroupMPI::createProcessGroupMPI()  ???:0
 9 0x0000000000c06e05 pybind11::cpp_function::initialize<torch::distributed::c10d::(anonymous namespace)::c10d_init(_object*, _object*)::{lambda(std::vector<int, std::allocator<int> >)#46}, c10::intrusive_ptr<c10d::ProcessGroupMPI, c10::detail::intrusive_target_default_null_type<c10d::ProcessGroupMPI> >, std::vector<int, std::allocator<int> >, pybind11::name, pybind11::scope, pybind11::sibling, pybind11::call_guard<pybind11::gil_scoped_release> >(torch::distributed::c10d::(anonymous namespace)::c10d_init(_object*, _object*)::{lambda(std::vector<int, std::allocator<int> >)#46}&&, c10::intrusive_ptr<c10d::ProcessGroupMPI, c10::detail::intrusive_target_default_null_type<c10d::ProcessGroupMPI> > (*)(std::vector<int, std::allocator<int> >), pybind11::name const&, pybind11::scope const&, pybind11::sibling const&, pybind11::call_guard<pybind11::gil_scoped_release> const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN()  init.cpp:0
10 0x00000000003ee25d pybind11::cpp_function::dispatcher()  :0
11 0x00000000005f5b39 PyCFunction_Call()  ???:0
12 0x00000000005f6706 _PyObject_MakeTpCall()  ???:0
13 0x0000000000571143 _PyEval_EvalFrameDefault()  ???:0
14 0x0000000000569d8a _PyEval_EvalCodeWithName()  ???:0
15 0x00000000005f60c3 _PyFunction_Vectorcall()  ???:0
16 0x000000000056cc92 _PyEval_EvalFrameDefault()  ???:0
17 0x0000000000569d8a _PyEval_EvalCodeWithName()  ???:0
18 0x00000000005f60c3 _PyFunction_Vectorcall()  ???:0
19 0x000000000056cc92 _PyEval_EvalFrameDefault()  ???:0
20 0x00000000005f5ee6 _PyFunction_Vectorcall()  ???:0
21 0x000000000056bab6 _PyEval_EvalFrameDefault()  ???:0
22 0x0000000000569d8a _PyEval_EvalCodeWithName()  ???:0
23 0x00000000005f60c3 _PyFunction_Vectorcall()  ???:0
24 0x000000000056bab6 _PyEval_EvalFrameDefault()  ???:0
25 0x0000000000569d8a _PyEval_EvalCodeWithName()  ???:0
26 0x00000000005f60c3 _PyFunction_Vectorcall()  ???:0
27 0x000000000056cc92 _PyEval_EvalFrameDefault()  ???:0
28 0x0000000000569d8a _PyEval_EvalCodeWithName()  ???:0
29 0x00000000005f60c3 _PyFunction_Vectorcall()  ???:0
30 0x000000000056cc92 _PyEval_EvalFrameDefault()  ???:0
31 0x0000000000569d8a _PyEval_EvalCodeWithName()  ???:0
32 0x000000000068e267 PyEval_EvalCode()  ???:0
33 0x000000000067d9b1 Py_CompileStringObject()  ???:0
34 0x000000000067da2f Py_CompileStringObject()  ???:0
35 0x000000000067dad1 Py_CompileStringObject()  ???:0
36 0x000000000067fbf7 PyRun_SimpleFileExFlags()  ???:0
37 0x00000000006b8082 Py_RunMain()  ???:0
38 0x00000000006b840d Py_BytesMain()  ???:0
39 0x0000000000024083 __libc_start_main()  ???:0
40 0x00000000005faa2e _start()  ???:0
```

### Versions

docker pull nvcr.io/nvidia/pytorch:23.11-py3 # a container of nvidia

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @fegin @XilunWu @wanchaol @fduwjj @wz337 @tianyu-l @wconstab @yf225