Repository: pytorch
Issue: DISABLED test_ddp_has_finalized (__main__.TestDistBackendWithSpawn)
Labels: triaged,skipped

Platforms: rocm, linux

ex https://ossci-raw-job-status.s3.amazonaws.com/log/13919698024
I think it started at https://github.com/pytorch/pytorch/pull/100773
```

2023-06-01T13:49:24.8049674Z distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized <- ../../../../opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/distributed/distributed_test.py [W Module.cpp:1349] Warning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (function operator())
2023-06-01T13:49:24.8050524Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
2023-06-01T13:49:24.8051003Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
2023-06-01T13:49:24.8052046Z [W reducer.cpp:1319] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-06-01T13:49:24.8053669Z [W reducer.cpp:1319] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-06-01T13:49:24.8054874Z [E ProcessGroupNCCL.cpp:456] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=4, OpType=ALLREDUCE, NumelIn=2048, NumelOut=2048, Timeout(ms)=5000) ran for 5219 milliseconds before timing out.
2023-06-01T13:49:24.8055652Z [E ProcessGroupNCCL.cpp:470] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
2023-06-01T13:49:24.8056300Z [E ProcessGroupNCCL.cpp:476] To avoid data inconsistency, we are taking the entire process down.
2023-06-01T13:49:24.8056991Z [E ProcessGroupNCCL.cpp:829] [Rank 1] NCCL watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=4, OpType=ALLREDUCE, NumelIn=2048, NumelOut=2048, Timeout(ms)=5000) ran for 5219 milliseconds before timing out.
2023-06-01T13:49:24.8057581Z SIGABRT(6), PID: 8353, Thread 8353: 
2023-06-01T13:49:24.8058243Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8b (0x7f0e4e3befab in /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libc10.so)
2023-06-01T13:49:24.8059049Z frame #1: <unknown function> + 0x14420 (0x7f0e70f76420 in /lib/x86_64-linux-gnu/libpthread.so.0)
2023-06-01T13:49:24.8059588Z frame #2: <unknown function> + 0x58d76 (0x7f0e0af4bd76 in /opt/rocm-5.4.2/lib/libhsa-runtime64.so.1)
2023-06-01T13:49:24.8060169Z frame #3: <unknown function> + 0x58c1e (0x7f0e0af4bc1e in /opt/rocm-5.4.2/lib/libhsa-runtime64.so.1)
2023-06-01T13:49:24.8060759Z frame #4: <unknown function> + 0x4e909 (0x7f0e0af41909 in /opt/rocm-5.4.2/lib/libhsa-runtime64.so.1)
2023-06-01T13:49:24.8061220Z frame #5: <unknown function> + 0xe3b5 (0x7f0e346cc3b5 in /opt/rocm/lib/libroctracer64.so.4)
2023-06-01T13:49:24.8061654Z frame #6: <unknown function> + 0x346493 (0x7f0e4ccba493 in /opt/rocm/hip/lib/libamdhip64.so.5)
2023-06-01T13:49:24.8062116Z frame #7: <unknown function> + 0xd89df (0x7f0e4ca4c9df in /opt/rocm/hip/lib/libamdhip64.so.5)
2023-06-01T13:49:24.8062585Z frame #8: hipEventSynchronize + 0x1da (0x7f0e4ca4f7ca in /opt/rocm/hip/lib/libamdhip64.so.5)
2023-06-01T13:49:24.8063240Z frame #9: <unknown function> + 0x1bdc673 (0x7f0e50049673 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch_hip.so)
2023-06-01T13:49:24.8064164Z frame #10: c10d::Logger::calculate_avg_time(long&, long&, c10d::Timer&, c10d::Timer::Event, c10d::Timer::Event) + 0x27 (0x7f0e5e262a47 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
2023-06-01T13:49:24.8064993Z frame #11: c10d::Logger::set_runtime_stats_and_log() + 0x5ef (0x7f0e5e26764f in /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
2023-06-01T13:49:24.8065719Z frame #12: <unknown function> + 0xc246ac (0x7f0e6950e6ac in /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
2023-06-01T13:49:24.8066419Z frame #13: <unknown function> + 0x3f5b5d (0x7f0e68cdfb5d in /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
2023-06-01T13:49:24.8066889Z frame #14: PyCFunction_Call + 0x52 (0x4f5652 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8067334Z frame #15: _PyObject_MakeTpCall + 0x3bb (0x4e0c8b in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8067735Z frame #16: /opt/conda/envs/py_3.8/bin/python() [0x4f53fd]
2023-06-01T13:49:24.8068153Z frame #17: _PyEval_EvalFrameDefault + 0x49a9 (0x4dc999 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8068589Z frame #18: _PyEval_EvalCodeWithName + 0x2f1 (0x4d6fb1 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8069047Z frame #19: _PyFunction_Vectorcall + 0x19c (0x4e807c in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8069446Z frame #20: /opt/conda/envs/py_3.8/bin/python() [0x4f5234]
2023-06-01T13:49:24.8069812Z frame #21: PyObject_Call + 0x24a (0x4f768a in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8070245Z frame #22: _PyEval_EvalFrameDefault + 0x1f7b (0x4d9f6b in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8070700Z frame #23: _PyEval_EvalCodeWithName + 0x2f1 (0x4d6fb1 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8071151Z frame #24: _PyFunction_Vectorcall + 0x19c (0x4e807c in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8071529Z frame #25: /opt/conda/envs/py_3.8/bin/python() [0x4f5234]
2023-06-01T13:49:24.8071916Z frame #26: PyObject_Call + 0x24a (0x4f768a in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8072361Z frame #27: _PyEval_EvalFrameDefault + 0x1f7b (0x4d9f6b in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8072789Z frame #28: _PyEval_EvalCodeWithName + 0x2f1 (0x4d6fb1 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8073235Z frame #29: _PyFunction_Vectorcall + 0x19c (0x4e807c in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8073632Z frame #30: /opt/conda/envs/py_3.8/bin/python() [0x4f5234]
2023-06-01T13:49:24.8074021Z frame #31: PyObject_Call + 0x24a (0x4f768a in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8074430Z frame #32: _PyEval_EvalFrameDefault + 0x1f7b (0x4d9f6b in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8074884Z frame #33: _PyEval_EvalCodeWithName + 0x2f1 (0x4d6fb1 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8075450Z frame #34: _PyObject_FastCallDict + 0x21b (0x4e03db in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8075865Z frame #35: _PyObject_Call_Prepend + 0x60 (0x4f1fd0 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8076270Z frame #36: /opt/conda/envs/py_3.8/bin/python() [0x5ab347]
2023-06-01T13:49:24.8076673Z frame #37: _PyObject_MakeTpCall + 0x3bb (0x4e0c8b in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8077122Z frame #38: _PyEval_EvalFrameDefault + 0x4907 (0x4dc8f7 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8077547Z frame #39: _PyEval_EvalCodeWithName + 0x2f1 (0x4d6fb1 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8077992Z frame #40: _PyFunction_Vectorcall + 0x19c (0x4e807c in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8078424Z frame #41: PyObject_Call + 0x24a (0x4f768a in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8078832Z frame #42: _PyEval_EvalFrameDefault + 0x1f7b (0x4d9f6b in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8079302Z frame #43: _PyEval_EvalCodeWithName + 0x2f1 (0x4d6fb1 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8079707Z frame #44: /opt/conda/envs/py_3.8/bin/python() [0x4f51bb]
2023-06-01T13:49:24.8080202Z frame #45: _PyEval_EvalFrameDefault + 0x399 (0x4d8389 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8080629Z frame #46: _PyEval_EvalCodeWithName + 0x2f1 (0x4d6fb1 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8081034Z frame #47: /opt/conda/envs/py_3.8/bin/python() [0x4f51bb]
2023-06-01T13:49:24.8081439Z frame #48: _PyEval_EvalFrameDefault + 0x399 (0x4d8389 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8081854Z frame #49: _PyFunction_Vectorcall + 0x106 (0x4e7fe6 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8082303Z frame #50: _PyEval_EvalFrameDefault + 0x6b2 (0x4d86a2 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8082745Z frame #51: _PyFunction_Vectorcall + 0x106 (0x4e7fe6 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8083150Z frame #52: /opt/conda/envs/py_3.8/bin/python() [0x4f5234]
2023-06-01T13:49:24.8083514Z frame #53: PyObject_Call + 0x24a (0x4f768a in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8083962Z frame #54: _PyEval_EvalFrameDefault + 0x1f7b (0x4d9f6b in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8084412Z frame #55: _PyFunction_Vectorcall + 0x106 (0x4e7fe6 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8084835Z frame #56: _PyEval_EvalFrameDefault + 0x6b2 (0x4d86a2 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8085287Z frame #57: _PyEval_EvalCodeWithName + 0x2f1 (0x4d6fb1 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8085733Z frame #58: _PyFunction_Vectorcall + 0x19c (0x4e807c in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8086182Z frame #59: _PyEval_EvalFrameDefault + 0x6b2 (0x4d86a2 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8086603Z frame #60: _PyFunction_Vectorcall + 0x106 (0x4e7fe6 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8087135Z frame #61: _PyEval_EvalFrameDefault + 0x399 (0x4d8389 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8087589Z frame #62: _PyEval_EvalCodeWithName + 0x2f1 (0x4d6fb1 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8088042Z frame #63: _PyFunction_Vectorcall + 0x19c (0x4e807c in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8088257Z 
2023-06-01T13:49:24.8088393Z SIGABRT(6), PID: 8353, Thread 8358: 
2023-06-01T13:49:24.8089062Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8b (0x7f0e4e3befab in /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libc10.so)
2023-06-01T13:49:24.8089728Z frame #1: <unknown function> + 0x14420 (0x7f0e70f76420 in /lib/x86_64-linux-gnu/libpthread.so.0)
2023-06-01T13:49:24.8090203Z frame #2: ioctl + 0xb (0x7f0e70d2a3ab in /lib/x86_64-linux-gnu/libc.so.6)
2023-06-01T13:49:24.8090745Z frame #3: <unknown function> + 0xe4840 (0x7f0e0afd7840 in /opt/rocm-5.4.2/lib/libhsa-runtime64.so.1)
2023-06-01T13:49:24.8091467Z frame #4: <unknown function> + 0xde525 (0x7f0e0afd1525 in /opt/rocm-5.4.2/lib/libhsa-runtime64.so.1)
2023-06-01T13:49:24.8092045Z frame #5: <unknown function> + 0x71de5 (0x7f0e0af64de5 in /opt/rocm-5.4.2/lib/libhsa-runtime64.so.1)
2023-06-01T13:49:24.8092598Z frame #6: <unknown function> + 0x55756 (0x7f0e0af48756 in /opt/rocm-5.4.2/lib/libhsa-runtime64.so.1)
2023-06-01T13:49:24.8093171Z frame #7: <unknown function> + 0x657ca (0x7f0e0af587ca in /opt/rocm-5.4.2/lib/libhsa-runtime64.so.1)
2023-06-01T13:49:24.8093743Z frame #8: <unknown function> + 0x27f7b (0x7f0e0af1af7b in /opt/rocm-5.4.2/lib/libhsa-runtime64.so.1)
2023-06-01T13:49:24.8094306Z frame #9: <unknown function> + 0x8609 (0x7f0e70f6a609 in /lib/x86_64-linux-gnu/libpthread.so.0)
2023-06-01T13:49:24.8094785Z frame #10: clone + 0x43 (0x7f0e70d35133 in /lib/x86_64-linux-gnu/libc.so.6)
2023-06-01T13:49:24.8095006Z 
2023-06-01T13:49:24.8095140Z SIGABRT(6), PID: 8353, Thread 8361: 
2023-06-01T13:49:24.8095803Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8b (0x7f0e4e3befab in /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libc10.so)
2023-06-01T13:49:24.8096724Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x152 (0x7f0e4e3bf4f2 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libc10.so)
2023-06-01T13:49:24.8097355Z frame #2: <unknown function> + 0x14420 (0x7f0e70f76420 in /lib/x86_64-linux-gnu/libpthread.so.0)
2023-06-01T13:49:24.8097870Z frame #3: gsignal + 0xcb (0x7f0e70c5900b in /lib/x86_64-linux-gnu/libc.so.6)
2023-06-01T13:49:24.8098349Z frame #4: abort + 0x12b (0x7f0e70c38859 in /lib/x86_64-linux-gnu/libc.so.6)
2023-06-01T13:49:24.8098749Z frame #5: <unknown function> + 0xb135a (0x7f0e3482735a in /opt/conda/envs/py_3.8/lib/libstdc++.so.6)
2023-06-01T13:49:24.8099216Z frame #6: <unknown function> + 0xb13c5 (0x7f0e348273c5 in /opt/conda/envs/py_3.8/lib/libstdc++.so.6)
2023-06-01T13:49:24.8099678Z frame #7: <unknown function> + 0xb134f (0x7f0e3482734f in /opt/conda/envs/py_3.8/lib/libstdc++.so.6)
2023-06-01T13:49:24.8100327Z frame #8: <unknown function> + 0x4930f0 (0x7f0e4e9000f0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch_hip.so)
2023-06-01T13:49:24.8100811Z frame #9: <unknown function> + 0xdbbf4 (0x7f0e34851bf4 in /opt/conda/envs/py_3.8/lib/libstdc++.so.6)
2023-06-01T13:49:24.8101372Z frame #10: <unknown function> + 0x8609 (0x7f0e70f6a609 in /lib/x86_64-linux-gnu/libpthread.so.0)
2023-06-01T13:49:24.8101878Z frame #11: clone + 0x43 (0x7f0e70d35133 in /lib/x86_64-linux-gnu/libc.so.6)
2023-06-01T13:49:24.8102098Z 
2023-06-01T13:49:24.8102229Z SIGABRT(6), PID: 8353, Thread 8363: 
2023-06-01T13:49:24.8102851Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8b (0x7f0e4e3befab in /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libc10.so)
2023-06-01T13:49:24.8103509Z frame #1: <unknown function> + 0x14420 (0x7f0e70f76420 in /lib/x86_64-linux-gnu/libpthread.so.0)
2023-06-01T13:49:24.8104021Z frame #2: __poll + 0x4f (0x7f0e70d2899f in /lib/x86_64-linux-gnu/libc.so.6)
2023-06-01T13:49:24.8104650Z frame #3: <unknown function> + 0x4996 (0x7f0e7089d996 in /opt/conda/envs/py_3.8/lib/python3.8/lib-dynload/select.cpython-38-x86_64-linux-gnu.so)
2023-06-01T13:49:24.8105093Z frame #4: /opt/conda/envs/py_3.8/bin/python() [0x4f5ab4]
2023-06-01T13:49:24.8105511Z frame #5: _PyEval_EvalFrameDefault + 0x6b2 (0x4d86a2 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8105971Z frame #6: _PyEval_EvalCodeWithName + 0x2f1 (0x4d6fb1 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8106395Z frame #7: _PyFunction_Vectorcall + 0x19c (0x4e807c in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8106845Z frame #8: _PyEval_EvalFrameDefault + 0x6b2 (0x4d86a2 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8107307Z frame #9: _PyEval_EvalCodeWithName + 0x2f1 (0x4d6fb1 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8107759Z frame #10: _PyFunction_Vectorcall + 0x19c (0x4e807c in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8108297Z frame #11: _PyEval_EvalFrameDefault + 0x49a9 (0x4dc999 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8108742Z frame #12: _PyFunction_Vectorcall + 0x106 (0x4e7fe6 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8109177Z frame #13: PyObject_Call + 0x24a (0x4f768a in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8109587Z frame #14: _PyEval_EvalFrameDefault + 0x1f7b (0x4d9f6b in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8110040Z frame #15: _PyFunction_Vectorcall + 0x106 (0x4e7fe6 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8110490Z frame #16: _PyEval_EvalFrameDefault + 0x6b2 (0x4d86a2 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8110935Z frame #17: _PyFunction_Vectorcall + 0x106 (0x4e7fe6 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8111364Z frame #18: _PyEval_EvalFrameDefault + 0x6b2 (0x4d86a2 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8111818Z frame #19: _PyFunction_Vectorcall + 0x106 (0x4e7fe6 in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8112220Z frame #20: /opt/conda/envs/py_3.8/bin/python() [0x4f5234]
2023-06-01T13:49:24.8112610Z frame #21: PyObject_Call + 0x24a (0x4f768a in /opt/conda/envs/py_3.8/bin/python)
2023-06-01T13:49:24.8113061Z frame #22: /opt/conda/envs/py_3.8/bin/python() [0x5bc827]
2023-06-01T13:49:24.8113449Z frame #23: /opt/conda/envs/py_3.8/bin/python() [0x5bc7d4]
2023-06-01T13:49:24.8113970Z frame #24: <unknown function> + 0x8609 (0x7f0e70f6a609 in /lib/x86_64-linux-gnu/libpthread.so.0)
2023-06-01T13:49:24.8114450Z frame #25: clone + 0x43 (0x7f0e70d35133 in /lib/x86_64-linux-gnu/libc.so.6)
2023-06-01T13:49:24.8114670Z 
2023-06-01T13:49:24.8114807Z SIGABRT(6), PID: 8353, Thread 8368: 
2023-06-01T13:49:24.8115450Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8b (0x7f0e4e3befab in /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libc10.so)
2023-06-01T13:49:24.8116118Z frame #1: <unknown function> + 0x14420 (0x7f0e70f76420 in /lib/x86_64-linux-gnu/libpthread.so.0)
2023-06-01T13:49:24.8116594Z frame #2: __poll + 0x4f (0x7f0e70d2899f in /lib/x86_64-linux-gnu/libc.so.6)
2023-06-01T13:49:24.8117116Z frame #3: <unknown function> + 0x98e5619 (0x7f0e33c03619 in /opt/rocm-5.4.2/lib/librccl.so.1)
2023-06-01T13:49:24.8117662Z frame #4: <unknown function> + 0x8609 (0x7f0e70f6a609 in /lib/x86_64-linux-gnu/libpthread.so.0)
2023-06-01T13:49:24.8118135Z frame #5: clone + 0x43 (0x7f0e70d35133 in /lib/x86_64-linux-gnu/libc.so.6)
2023-06-01T13:49:24.8118354Z 
2023-06-01T13:49:24.8118610Z SIGABRT(6), PID: 8353, Thread 8369: 
2023-06-01T13:49:24.8119258Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8b (0x7f0e4e3befab in /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libc10.so)
2023-06-01T13:49:24.8119915Z frame #1: <unknown function> + 0x14420 (0x7f0e70f76420 in /lib/x86_64-linux-gnu/libpthread.so.0)
2023-06-01T13:49:24.8120430Z frame #2: pthread_cond_wait + 0x216 (0x7f0e70f71376 in /lib/x86_64-linux-gnu/libpthread.so.0)
2023-06-01T13:49:24.8120979Z frame #3: <unknown function> + 0x98e4164 (0x7f0e33c02164 in /opt/rocm-5.4.2/lib/librccl.so.1)
2023-06-01T13:49:24.8121524Z frame #4: <unknown function> + 0x8609 (0x7f0e70f6a609 in /lib/x86_64-linux-gnu/libpthread.so.0)
2023-06-01T13:49:24.8122028Z frame #5: clone + 0x43 (0x7f0e70d35133 in /lib/x86_64-linux-gnu/libc.so.6)
2023-06-01T13:49:24.8122245Z 
2023-06-01T13:49:24.8122662Z [2023-06-01 10:11:09,272] torch.testing._internal.common_distributed: [ERROR] Encountered error while trying to get traceback for process 1: [Errno 32] Broken pipe
```

This test was disabled because it is failing on master ([recent examples](http://torch-ci.com/failure/distributed%2Ffsdp%2Ftest_fsdp_core.py%3A%3ATestParityWithDDP%3A%3Atest_delayed_optim_step_offload_true_shard_grad_op)).

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang