Repository: pytorch
Issue: [Distributed: RPC] Failed to initialize RPC with >18 workers
Labels: oncall: distributed,triaged,module: rpc

### 🐛 Describe the bug

Failed to initialize RPC with >18 workers. Here is a minimal script:

```python
# rpc_test.py

import os
import random

import numpy as np
import torch

import torch.distributed.rpc as rpc


def worker_init():
    rank = int(os.environ['RANK'])

    random.seed(rank)
    np.random.seed(rank)
    torch.manual_seed(rank)
    print(f'Rank {rank}')


def main():
    rank = int(os.environ['RANK'])
    world_size = int(os.environ['WORLD_SIZE'])

    rpc.init_rpc(name=f'worker{rank}', rank=rank, world_size=world_size)
    worker_init()

    # no-op

    rpc.shutdown()


if __name__ == '__main__':
    main()
```

Run in command line:

```console
$ torchrun --nnode 1 --nproc_per_node 18 rpc_test.py
WARNING:torch.distributed.run:
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
Rank 0
Rank 1
Rank 2
Rank 3
Rank 4
Rank 5
Rank 6
Rank 7
Rank 8
Rank 9
Rank 10
Rank 11
Rank 12
Rank 13
Rank 14
Rank 15
Rank 17
Rank 16
```

It works for me for 18 workers but raises errors when I change the proc number larger than 18 (e.g. 19)

Output:

```console
$ torchrun --nnode 1 --nproc_per_node 19 rpc_test.py
WARNING:torch.distributed.run:
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
[W tensorpipe_agent.cpp:916] RPC agent for worker10 encountered error when sending outgoing request #0 to worker0: connect: Resource temporarily unavailable (this error originated at tensorpipe/common/socket.cc:114)
[W tensorpipe_agent.cpp:916] RPC agent for worker2 encountered error when sending outgoing request #0 to worker0: connect: Resource temporarily unavailable (this error originated at tensorpipe/common/socket.cc:114)
Traceback (most recent call last):
  File "rpc_test.py", line 33, in <module>
[W tensorpipe_agent.cpp:916] RPC agent for worker16 encountered error when sending outgoing request #0 to worker0: connect: Resource temporarily unavailable (this error originated at tensorpipe/common/socket.cc:114)
    main()
  File "rpc_test.py", line 24, in main
    rpc.init_rpc(name=f'worker{rank}', rank=rank, world_size=world_size)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 196, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 231, in _init_rpc_backend
    rpc_agent = backend_registry.init_backend(
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 101, in init_backend
Traceback (most recent call last):
  File "rpc_test.py", line 33, in <module>
    return backend.value.init_backend_handler(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 359, in _tensorpipe_init_backend_handler
Traceback (most recent call last):
  File "rpc_test.py", line 33, in <module>
    main()
  File "rpc_test.py", line 24, in main
        api._all_gather(None, timeout=rpc_backend_options.rpc_timeout)
rpc.init_rpc(name=f'worker{rank}', rank=rank, world_size=world_size)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 196, in init_rpc
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper
        _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)return func(*args, **kwargs)

  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 231, in _init_rpc_backend
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 222, in _all_gather
    main()
  File "rpc_test.py", line 24, in main
            rpc_agent = backend_registry.init_backend(rpc_sync(rpc.init_rpc(name=f'worker{rank}', rank=rank, world_size=world_size)


  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 101, in init_backend
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 196, in init_rpc
        return backend.value.init_backend_handler(*args, **kwargs)return func(*args, **kwargs)

  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 359, in _tensorpipe_init_backend_handler
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 800, in rpc_sync
[W tensorpipe_agent.cpp:916] RPC agent for worker14 encountered error when sending outgoing request #0 to worker0: connect: Resource temporarily unavailable (this error originated at tensorpipe/common/socket.cc:114)
        _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)api._all_gather(None, timeout=rpc_backend_options.rpc_timeout)

  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 231, in _init_rpc_backend
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper
    return func(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 222, in _all_gather
    rpc_agent = backend_registry.init_backend(
      File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 101, in init_backend
rpc_sync(    
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper
return fut.wait()
        return backend.value.init_backend_handler(*args, **kwargs)return func(*args, **kwargs)
RuntimeError
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 359, in _tensorpipe_init_backend_handler
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 800, in rpc_sync
: connect: Resource temporarily unavailable (this error originated at tensorpipe/common/socket.cc:114)
    api._all_gather(None, timeout=rpc_backend_options.rpc_timeout)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper
    return func(*args, **kwargs)    
return fut.wait()
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 222, in _all_gather
RuntimeError: connect: Resource temporarily unavailable (this error originated at tensorpipe/common/socket.cc:114)
    rpc_sync(
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper
    return func(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 800, in rpc_sync
Traceback (most recent call last):
    return fut.wait()
  File "rpc_test.py", line 33, in <module>
RuntimeError: connect: Resource temporarily unavailable (this error originated at tensorpipe/common/socket.cc:114)
    main()
  File "rpc_test.py", line 24, in main
    rpc.init_rpc(name=f'worker{rank}', rank=rank, world_size=world_size)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 196, in init_rpc
[W tensorpipe_agent.cpp:916] RPC agent for worker4 encountered error when sending outgoing request #0 to worker0: connect: Resource temporarily unavailable (this error originated at tensorpipe/common/socket.cc:114)
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 231, in _init_rpc_backend
    rpc_agent = backend_registry.init_backend(
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 101, in init_backend
[W tensorpipe_agent.cpp:916] RPC agent for worker8 encountered error when sending outgoing request #0 to worker0: connect: Resource temporarily unavailable (this error originated at tensorpipe/common/socket.cc:114)
    return backend.value.init_backend_handler(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 359, in _tensorpipe_init_backend_handler
    api._all_gather(None, timeout=rpc_backend_options.rpc_timeout)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper
    return func(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 222, in _all_gather
    rpc_sync(
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper
    return func(*args, **kwargs)Traceback (most recent call last):

  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 800, in rpc_sync
  File "rpc_test.py", line 33, in <module>
Traceback (most recent call last):
  File "rpc_test.py", line 33, in <module>
    return fut.wait()
RuntimeError: connect: Resource temporarily unavailable (this error originated at tensorpipe/common/socket.cc:114)
    main()
  File "rpc_test.py", line 24, in main
    rpc.init_rpc(name=f'worker{rank}', rank=rank, world_size=world_size)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 196, in init_rpc
    main()
  File "rpc_test.py", line 24, in main
        _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)rpc.init_rpc(name=f'worker{rank}', rank=rank, world_size=world_size)

  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 231, in _init_rpc_backend
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 196, in init_rpc
        _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)rpc_agent = backend_registry.init_backend(

  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 231, in _init_rpc_backend
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 101, in init_backend
        rpc_agent = backend_registry.init_backend(return backend.value.init_backend_handler(*args, **kwargs)

  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 101, in init_backend
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 359, in _tensorpipe_init_backend_handler
    return backend.value.init_backend_handler(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 359, in _tensorpipe_init_backend_handler
        api._all_gather(None, timeout=rpc_backend_options.rpc_timeout)api._all_gather(None, timeout=rpc_backend_options.rpc_timeout)

  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper
        return func(*args, **kwargs)return func(*args, **kwargs)

  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 222, in _all_gather
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 222, in _all_gather
        rpc_sync(rpc_sync(

  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper
    return func(*args, **kwargs)    
return func(*args, **kwargs)  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 800, in rpc_sync

  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 800, in rpc_sync
    return fut.wait()
RuntimeError: connect: Resource temporarily unavailable (this error originated at tensorpipe/common/socket.cc:114)
    return fut.wait()
RuntimeError: connect: Resource temporarily unavailable (this error originated at tensorpipe/common/socket.cc:114)
[W tensorpipe_agent.cpp:530] RPC agent for worker0 encountered error when accepting incoming pipe: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker14: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker10: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker15: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker9: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker8: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker1: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker17: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker6: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker11: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker13: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker7: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker3: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker5: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker4: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker2: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker16: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker12: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:726] RPC agent for worker0 encountered error when reading incoming request from worker18: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:916] RPC agent for worker18 encountered error when sending outgoing request #0 to worker0: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:916] RPC agent for worker5 encountered error when sending outgoing request #0 to worker0: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:940] RPC agent for worker17 encountered error when reading incoming response from worker0: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:916] RPC agent for worker3 encountered error when sending outgoing request #0 to worker0: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:940] RPC agent for worker9 encountered error when reading incoming response from worker0: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:916] RPC agent for worker15 encountered error when sending outgoing request #0 to worker0: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:940] RPC agent for worker1 encountered error when reading incoming response from worker0: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:940] RPC agent for worker13 encountered error when reading incoming response from worker0: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:940] RPC agent for worker7 encountered error when reading incoming response from worker0: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:916] RPC agent for worker12 encountered error when sending outgoing request #0 to worker0: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
[W tensorpipe_agent.cpp:940] RPC agent for worker11 encountered error when reading incoming response from worker0: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
Traceback (most recent call last):
  File "rpc_test.py", line 33, in <module>
Traceback (most recent call last):
  File "rpc_test.py", line 33, in <module>
Traceback (most recent call last):
Traceback (most recent call last):
  File "rpc_test.py", line 33, in <module>
  File "rpc_test.py", line 33, in <module>
Traceback (most recent call last):
  File "rpc_test.py", line 33, in <module>
        main()main()

  File "rpc_test.py", line 24, in main
  File "rpc_test.py", line 24, in main
    rpc.init_rpc(name=f'worker{rank}', rank=rank, world_size=world_size)    
rpc.init_rpc(name=f'worker{rank}', rank=rank, world_size=world_size)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 196, in init_rpc
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 196, in init_rpc
    main()    main()
  File "rpc_test.py", line 24, in main

  File "rpc_test.py", line 24, in main
    rpc.init_rpc(name=f'worker{rank}', rank=rank, world_size=world_size)
          File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 196, in init_rpc
_init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)rpc.init_rpc(name=f'worker{rank}', rank=rank, world_size=world_size)

      File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 231, in _init_rpc_backend
_init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 196, in init_rpc

  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 231, in _init_rpc_backend
    rpc_agent = backend_registry.init_backend(    
_init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)      File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 101, in init_backend
_init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)

  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 231, in _init_rpc_backend
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 231, in _init_rpc_backend
    rpc_agent = backend_registry.init_backend(
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 101, in init_backend
    main()    
return backend.value.init_backend_handler(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 359, in _tensorpipe_init_backend_handler
    rpc_agent = backend_registry.init_backend(  File "rpc_test.py", line 24, in main

          File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 101, in init_backend
return backend.value.init_backend_handler(*args, **kwargs)rpc_agent = backend_registry.init_backend(

  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 359, in _tensorpipe_init_backend_handler
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 101, in init_backend
    return backend.value.init_backend_handler(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 359, in _tensorpipe_init_backend_handler
    return backend.value.init_backend_handler(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 359, in _tensorpipe_init_backend_handler
    api._all_gather(None, timeout=rpc_backend_options.rpc_timeout)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper
    return func(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 222, in _all_gather
    api._all_gather(None, timeout=rpc_backend_options.rpc_timeout)    
api._all_gather(None, timeout=rpc_backend_options.rpc_timeout)  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper

    api._all_gather(None, timeout=rpc_backend_options.rpc_timeout)  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper

  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper
        rpc_sync(
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper
rpc.init_rpc(name=f'worker{rank}', rank=rank, world_size=world_size)            return func(*args, **kwargs)return func(*args, **kwargs)
return func(*args, **kwargs)

  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 222, in _all_gather
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 222, in _all_gather

  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 222, in _all_gather
    return func(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 800, in rpc_sync
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 196, in init_rpc
    rpc_sync(    
rpc_sync(  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper

  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper
    rpc_sync(
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper
        return func(*args, **kwargs)return func(*args, **kwargs)

  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 800, in rpc_sync
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 800, in rpc_sync
    return func(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 800, in rpc_sync
    return fut.wait()
RuntimeError: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
    return fut.wait()
    return fut.wait()
RuntimeError: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)RuntimeError
: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 231, in _init_rpc_backend
    return fut.wait()
RuntimeError: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
    rpc_agent = backend_registry.init_backend(
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 101, in init_backend
    return backend.value.init_backend_handler(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 359, in _tensorpipe_init_backend_handler
    api._all_gather(None, timeout=rpc_backend_options.rpc_timeout)
Traceback (most recent call last):
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper
  File "rpc_test.py", line 33, in <module>
    main()
  File "rpc_test.py", line 24, in main
    rpc.init_rpc(name=f'worker{rank}', rank=rank, world_size=world_size)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 196, in init_rpc
    return func(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 222, in _all_gather
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 231, in _init_rpc_backend
    rpc_agent = backend_registry.init_backend(
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 101, in init_backend
    return backend.value.init_backend_handler(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 359, in _tensorpipe_init_backend_handler
    api._all_gather(None, timeout=rpc_backend_options.rpc_timeout)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper
    rpc_sync(
      File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper
return func(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 222, in _all_gather
[W tensorpipe_agent.cpp:940] RPC agent for worker6 encountered error when reading incoming response from worker0: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
    rpc_sync(
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper
    return func(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 800, in rpc_sync
    return func(*args, **kwargs)    return fut.wait()

RuntimeError: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 800, in rpc_sync
Traceback (most recent call last):
  File "rpc_test.py", line 33, in <module>
    main()
  File "rpc_test.py", line 24, in main
    return fut.wait()    
rpc.init_rpc(name=f'worker{rank}', rank=rank, world_size=world_size)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 196, in init_rpc
RuntimeError: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 231, in _init_rpc_backend
    rpc_agent = backend_registry.init_backend(
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 101, in init_backend
    return backend.value.init_backend_handler(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 359, in _tensorpipe_init_backend_handler
    api._all_gather(None, timeout=rpc_backend_options.rpc_timeout)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper
    return func(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 222, in _all_gather
    rpc_sync(
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper
    return func(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 800, in rpc_sync
    return fut.wait()
RuntimeError: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
Traceback (most recent call last):
  File "rpc_test.py", line 33, in <module>
    main()
  File "rpc_test.py", line 24, in main
    rpc.init_rpc(name=f'worker{rank}', rank=rank, world_size=world_size)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 196, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 231, in _init_rpc_backend
    rpc_agent = backend_registry.init_backend(
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 101, in init_backend
    return backend.value.init_backend_handler(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 359, in _tensorpipe_init_backend_handler
    api._all_gather(None, timeout=rpc_backend_options.rpc_timeout)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper
    return func(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 222, in _all_gather
    rpc_sync(
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper
Traceback (most recent call last):
  File "rpc_test.py", line 33, in <module>
    return func(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 800, in rpc_sync
    return fut.wait()
RuntimeError: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
    main()
  File "rpc_test.py", line 24, in main
    rpc.init_rpc(name=f'worker{rank}', rank=rank, world_size=world_size)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 196, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 231, in _init_rpc_backend
    rpc_agent = backend_registry.init_backend(
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 101, in init_backend
    return backend.value.init_backend_handler(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 359, in _tensorpipe_init_backend_handler
    api._all_gather(None, timeout=rpc_backend_options.rpc_timeout)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper
    return func(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 222, in _all_gather
    rpc_sync(
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper
    return func(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 800, in rpc_sync
    return fut.wait()
RuntimeError: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
Traceback (most recent call last):
Traceback (most recent call last):
  File "rpc_test.py", line 33, in <module>
  File "rpc_test.py", line 33, in <module>
    main()
  File "rpc_test.py", line 24, in main
    rpc.init_rpc(name=f'worker{rank}', rank=rank, world_size=world_size)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 196, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 231, in _init_rpc_backend
    rpc_agent = backend_registry.init_backend(
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 101, in init_backend
    return backend.value.init_backend_handler(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 359, in _tensorpipe_init_backend_handler
    api._all_gather(None, timeout=rpc_backend_options.rpc_timeout)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper
    return func(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 222, in _all_gather
    main()    rpc_sync(

  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper
  File "rpc_test.py", line 24, in main
    return func(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 800, in rpc_sync
    return fut.wait()
RuntimeError: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
    rpc.init_rpc(name=f'worker{rank}', rank=rank, world_size=world_size)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 196, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 231, in _init_rpc_backend
    rpc_agent = backend_registry.init_backend(
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 101, in init_backend
    return backend.value.init_backend_handler(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 359, in _tensorpipe_init_backend_handler
    api._all_gather(None, timeout=rpc_backend_options.rpc_timeout)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper
    return func(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 222, in _all_gather
    rpc_sync(
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper
    return func(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 800, in rpc_sync
    return fut.wait()
RuntimeError: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
Traceback (most recent call last):
  File "rpc_test.py", line 33, in <module>
    main()
  File "rpc_test.py", line 24, in main
    rpc.init_rpc(name=f'worker{rank}', rank=rank, world_size=world_size)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 196, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 231, in _init_rpc_backend
    rpc_agent = backend_registry.init_backend(
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 101, in init_backend
    return backend.value.init_backend_handler(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 359, in _tensorpipe_init_backend_handler
    api._all_gather(None, timeout=rpc_backend_options.rpc_timeout)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper
    return func(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 222, in _all_gather
    rpc_sync(
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 83, in wrapper
    return func(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 800, in rpc_sync
    return fut.wait()
RuntimeError: async error on socket: Connection reset by peer (this error originated at tensorpipe/transport/shm/connection_impl.cc:187)
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 17244 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 1 (pid: 17245) of binary: /home/PanXuehai/Miniconda3/envs/torchopt/bin/python
Traceback (most recent call last):
  File "/home/PanXuehai/Miniconda3/envs/torchopt/bin/torchrun", line 33, in <module>
    sys.exit(load_entry_point('torch==1.12.1', 'console_scripts', 'torchrun')())
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper
    return f(*args, **kwargs)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/run.py", line 761, in main
    run(args)
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/run.py", line 752, in run
    elastic_launch(
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 131, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/PanXuehai/Miniconda3/envs/torchopt/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 245, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
rpc_test.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2022-09-25_22:37:55
  host      : BIGAI-PanXuehai.localdomain
  rank      : 2 (local_rank: 2)
  exitcode  : 1 (pid: 17246)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time      : 2022-09-25_22:37:55
  host      : BIGAI-PanXuehai.localdomain
  rank      : 3 (local_rank: 3)
  exitcode  : 1 (pid: 17247)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
  time      : 2022-09-25_22:37:55
  host      : BIGAI-PanXuehai.localdomain
  rank      : 4 (local_rank: 4)
  exitcode  : 1 (pid: 17248)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[4]:
  time      : 2022-09-25_22:37:55
  host      : BIGAI-PanXuehai.localdomain
  rank      : 5 (local_rank: 5)
  exitcode  : 1 (pid: 17249)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[5]:
  time      : 2022-09-25_22:37:55
  host      : BIGAI-PanXuehai.localdomain
  rank      : 6 (local_rank: 6)
  exitcode  : 1 (pid: 17250)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[6]:
  time      : 2022-09-25_22:37:55
  host      : BIGAI-PanXuehai.localdomain
  rank      : 7 (local_rank: 7)
  exitcode  : 1 (pid: 17251)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[7]:
  time      : 2022-09-25_22:37:55
  host      : BIGAI-PanXuehai.localdomain
  rank      : 8 (local_rank: 8)
  exitcode  : 1 (pid: 17252)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[8]:
  time      : 2022-09-25_22:37:55
  host      : BIGAI-PanXuehai.localdomain
  rank      : 9 (local_rank: 9)
  exitcode  : 1 (pid: 17253)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[9]:
  time      : 2022-09-25_22:37:55
  host      : BIGAI-PanXuehai.localdomain
  rank      : 10 (local_rank: 10)
  exitcode  : 1 (pid: 17254)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[10]:
  time      : 2022-09-25_22:37:55
  host      : BIGAI-PanXuehai.localdomain
  rank      : 11 (local_rank: 11)
  exitcode  : 1 (pid: 17256)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[11]:
  time      : 2022-09-25_22:37:55
  host      : BIGAI-PanXuehai.localdomain
  rank      : 12 (local_rank: 12)
  exitcode  : 1 (pid: 17259)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[12]:
  time      : 2022-09-25_22:37:55
  host      : BIGAI-PanXuehai.localdomain
  rank      : 13 (local_rank: 13)
  exitcode  : 1 (pid: 17264)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[13]:
  time      : 2022-09-25_22:37:55
  host      : BIGAI-PanXuehai.localdomain
  rank      : 14 (local_rank: 14)
  exitcode  : 1 (pid: 17267)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[14]:
  time      : 2022-09-25_22:37:55
  host      : BIGAI-PanXuehai.localdomain
  rank      : 15 (local_rank: 15)
  exitcode  : 1 (pid: 17268)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[15]:
  time      : 2022-09-25_22:37:55
  host      : BIGAI-PanXuehai.localdomain
  rank      : 16 (local_rank: 16)
  exitcode  : 1 (pid: 17269)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[16]:
  time      : 2022-09-25_22:37:55
  host      : BIGAI-PanXuehai.localdomain
  rank      : 17 (local_rank: 17)
  exitcode  : 1 (pid: 17271)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[17]:
  time      : 2022-09-25_22:37:55
  host      : BIGAI-PanXuehai.localdomain
  rank      : 18 (local_rank: 18)
  exitcode  : 1 (pid: 17273)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2022-09-25_22:37:55
  host      : BIGAI-PanXuehai.localdomain
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 17245)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
```

Different machines report different maximum worker numbers on my side.

- single-socket 16C32T CPU: raise error 19+ procs
- single-socket 6C12T CPU: raises error 9+ procs
- dual-socket 22C44T CPU (44C88T in total): raise error 9+ procs

### Versions

```
PyTorch version: 1.12.1
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (conda-forge gcc 10.4.0-16) 10.4.0
Clang version: 10.0.1 
CMake version: version 3.22.1
Libc version: glibc-2.31

Python version: 3.8.12 | packaged by conda-forge | (default, Jan 30 2022, 23:42:07)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.10
Is CUDA available: True
CUDA runtime version: 11.6.124
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3070
Nvidia driver version: 516.94
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] functorch==0.2.1
[pip3] mypy==0.910
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.23.1
[pip3] torch==1.12.1
[pip3] torchopt==0.5.1.dev66+gd738101
[pip3] torchvision==0.13.1
[pip3] torchviz==0.0.2
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.6.0               habf752d_9  
[conda] functorch                 0.2.1                    pypi_0  
[conda] libblas                   3.9.0            12_linux64_mkl  
[conda] libcblas                  3.9.0            12_linux64_mkl  
[conda] liblapack                 3.9.0            12_linux64_mkl  
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py38h7f8727e_0  
[conda] mkl_fft                   1.3.1            py38hd3c417c_0  
[conda] mkl_random                1.2.2            py38h51133e4_0  
[conda] numpy                     1.23.1           py38h6c91a56_0  
[conda] numpy-base                1.23.1           py38ha15fc14_0  
[conda] pytorch                   1.12.1          py3.8_cuda11.6_cudnn8.3.2_0  
[conda] pytorch-mutex             1.0                        cuda  
[conda] torchopt                  0.5.1.dev66+gd738101          pypi_0  
[conda] torchvision               0.13.1               py38_cu116  
[conda] torchviz                  0.0.2                    pypi_0
```

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501 @jjlilley @mrzzd