Repository: pytorch
Issue: All thread stack trace dumps interleave output in distributed tests, giving garbled output
Labels: oncall: distributed,triaged

As a result, I get garbled output like this (the extra newlines are an artifact of copy pasting from a screen session, but the random ordering is mine):

```
[5] pthread_cond_wait+0x1fc(0x7fa8faedc2fc) in /lib64/libpthread.so.0                                                                       
[6] std::condition_variable::wait(std::unique_lock<std::mutex>&)+0x9(0x7fa8d11da4cb) in /data/users/ezyang/pytorch-tmp2-env/lib/libstdc++.so
.6                                                                                                                                          
[7] torch::distributed::rpc::ProcessGroupAgent::listenLoopInternal()+0x239(0x7f72dc9f7423) in /data/users/ezyang/pytorch-tmp2/torch/lib/libt
orch_python.so                                                                                                                              
[15] ???+0x511ea5b(0x7f300511ea5b) in /data/users/ezyang/pytorch-tmp2/torch/lib/libtorch_cpu.so                                             
[2] ???(0x7fa8e4ff6014) in /data/users/ezyang/pytorch-tmp2/torch/lib/libtorch_python.so                                                     
[8] torch::distributed::rpc::ProcessGroupAgent::listenLoop()+0x22(0x7f72dc9f6eb6) in /data/users/ezyang/pytorch-tmp2/torch/lib/libtorch_pyth
on.so???                                                                                                                                    
([160x7fb08be8e9ed] ) in /data/users/ezyang/pytorch-tmp2/torch/lib/libtorch_cpu.so                                                          
[1] torch::distributed::rpc::ProcessGroupAgent::pollTimedOutRPCs()+0x129(0x7fa8e4ff4a97) in /data/users/ezyang/pytorch-tmp2/torch/lib/libtor
ch_python.so                                                                                                                                
[9] void std::__invoke_impl<void, void (torch::distributed::rpc::ProcessGroupAgent::*)(), torch::distributed::rpc::ProcessGroupAgent*>(std::
__invoke_memfun_deref, void (torch::distributed::rpc::ProcessGroupAgent::*&&)(), torch::distributed::rpc::ProcessGroupAgent*&&)+0x67(0x7f72d
ca06858) in /data/users/ezyang/pytorch-tmp2/torch/lib/libtorch_python.so                                                                    
[17] void std::__invoke_impl<void, void (torch::distributed::rpc::ProcessGroupAgent::*)(), torch::distributed::rpc::ProcessGroupAgent*>(std:
:__invoke_memfun_deref, void (torch::distributed::rpc::ProcessGroupAgent::*&&)(), torch::distributed::rpc::ProcessGroupAgent*&&)+0x67(0x7fa8
e5003858) in /data/users/ezyang/pytorch-tmp2/torch/lib/libtorch_python.so                                                                   
[std::__invoke_result<void (torch::distributed::rpc::ProcessGroupAgent::*)(), torch::distributed::rpc::ProcessGroupAgent*>::type std::__invo
ke<void (torch::distributed::rpc::ProcessGroupAgent::*)(), torch::distributed::rpc::ProcessGroupAgent*>(void (torch::distributed::rpc::Proce
ssGroupAgent::*&&)(), torch::distributed::rpc::ProcessGroupAgent*&&)10+] 0x37(0x7f72dca02815) in /data/users/ezyang/pytorch-tmp2/torch/lib/l
ibtorch_python.so                                                                                                                           
[18] ???+0x511ee08(0x7f300511ee08) in /data/users/ezyang/pytorch-tmp2/torch/lib/libtorch_cpu.so                                             
[3] ???(0x7fb08be8ea5b) in /data/users/ezyang/pytorch-tmp2/torch/lib/libtorch_cpu.so                                                        
[2] std::__invoke_result<void (torch::distributed::rpc::ProcessGroupAgent::*)(), torch::distributed::rpc::ProcessGroupAgent*>::type std::__i
nvoke<void (torch::distributed::rpc::ProcessGroupAgent::*)(), torch::distributed::rpc::ProcessGroupAgent*>(void (torch::distributed::rpc::Pr
ocessGroupAgent::*&&)(), torch::distributed::rpc::ProcessGroupAgent*&&)decltype (__invoke((_S_declval<0ul>)(), (_S_declval<1ul>)())) std::th
read::_Invoker<std::tuple<void (torch::distributed::rpc::ProcessGroupAgent::*)(), torch::distributed::rpc::ProcessGroupAgent*> >::_M_invoke<
0ul, 1ul>(std::_Index_tuple<0ul, 1ul>)++0x370x43((0x7fa8e4fff8150x7f72dca14c23)) in  in /data/users/ezyang/pytorch-tmp2/torch/lib/libtorch_p
ython.so/data/users/ezyang/pytorch-tmp2/torch/lib/libtorch_python.so
```

There should be an option to pipe the logs for each subprocess separately and print them one by one, or perhaps interleave the outputs on a line by line basis while also annotating them with what subprocess produced them (although this would still be pretty unreadable). 

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @agolynski @SciPioneer @H-Huang @mrzzd @cbalioglu @gcramer23 @driazati