Repository: pytorch
Issue: Building fails on a node with libibverbs installed with no Infiniband hardware present on RHEL8.
Labels: oncall: distributed,module: tensorpipe

## 🐛 Bug

The test `distributed/pipeline/sync/test_pipe.py` fails on a RHEL8 machine that has libibverbs installed but without any Infiniband hardware:

````
Executing ['/opt/uio/modules/rhel8/easybuild/software/Python/3.8.6-GCCcore-10.2.0/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_pipe.py', '-v'] ... [2021-06-23
 11:04:23.618580]
[...]
RuntimeError: In operator() at tensorpipe/common/ibv.h:151 "": Cannot allocate memory
distributed/pipeline/sync/test_transparency.py::test_simple_linears libi40iw-i40iw_ucreate_cq: failed to pin memory for CQ
[...]
$ rpm -qa | grep libibverbs
libibverbs-32.0-4.el8.x86_64
````

## To Reproduce

Steps to reproduce the behavior:

1. Install libibverbs on system.
2. Build pytorch from source.
3. Encounter test failure.

````
============================================================================= 4 passed in 0.04s ==============================================================================
Running distributed/pipeline/sync/test_pipe ... [2021-06-23 11:04:23.618493]
Executing ['/opt/uio/modules/rhel8/easybuild/software/Python/3.8.6-GCCcore-10.2.0/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_pipe.py', '-v'] ... [2021-06-23
 11:04:23.618580]
============================================================================ test session starts =============================================================================
platform linux -- Python 3.8.6, pytest-6.1.1, py-1.9.0, pluggy-0.13.1 -- /opt/uio/modules/rhel8/easybuild/software/Python/3.8.6-GCCcore-10.2.0/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/.hypothesis/examples')
torch: 1.9.0
rootdir: /tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch, configfile: pytest.ini
plugins: hypothesis-5.41.5
collecting ... collected 38 items

distributed/pipeline/sync/test_pipe.py::test_pipe_without_rpc PASSED                                                                                                   [  2%]
distributed/pipeline/sync/test_pipe.py::test_parameters libi40iw-i40iw_ucreate_cq: failed to pin memory for CQ
ERROR                                                                                                          [  5%]
distributed/pipeline/sync/test_pipe.py::test_public_attrs ERROR                                                                                                        [  7%]
distributed/pipeline/sync/test_pipe.py::test_sequential_like ERROR                                                                                                     [ 10%]
distributed/pipeline/sync/test_pipe.py::test_chunks_less_than_1 ERROR                                                                                                  [ 13%]
distributed/pipeline/sync/test_pipe.py::test_batch_size_indivisible ERROR                                                                                              [ 15%]
distributed/pipeline/sync/test_pipe.py::test_batch_size_small ERROR                                                                                                    [ 18%]
distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode ERROR                                                                                                     [ 21%]
distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_invalid ERROR                                                                                             [ 23%]
distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_when_chunks_1 ERROR                                                                                       [ 26%]
distributed/pipeline/sync/test_pipe.py::test_checkpoint_eval ERROR                                                                                                     [ 28%]
distributed/pipeline/sync/test_pipe.py::test_checkpoint_non_float_input ERROR                                                                                          [ 31%]
distributed/pipeline/sync/test_pipe.py::test_no_grad ERROR                                                                                                             [ 34%]
distributed/pipeline/sync/test_pipe.py::test_exception ERROR                                                                                                           [ 36%]
distributed/pipeline/sync/test_pipe.py::test_exception_early_stop_asap ERROR                                                                                           [ 39%]
distributed/pipeline/sync/test_pipe.py::test_nested_input ERROR                                                                                                        [ 42%]
distributed/pipeline/sync/test_pipe.py::test_input_pair ERROR                                                                                                          [ 44%]
distributed/pipeline/sync/test_pipe.py::test_input_singleton ERROR                                                                                                     [ 47%]
distributed/pipeline/sync/test_pipe.py::test_input_varargs ERROR                                                                                                       [ 50%]
distributed/pipeline/sync/test_pipe.py::test_non_tensor ERROR                                                                                                          [ 52%]
distributed/pipeline/sync/test_pipe.py::test_non_tensor_sequence ERROR                                                                                                 [ 55%]
distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[never] ERROR                                                                                          [ 57%]
distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[always] ERROR                                                                                         [ 60%]
distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[except_last] ERROR                                                                                    [ 63%]
distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[never] ERROR                                                                                   [ 65%]
distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[always] ERROR                                                                                  [ 68%]
distributed/pipeline/sync/test_pipe.py::test_devices ERROR                                                                                                             [ 71%]
distributed/pipeline/sync/test_pipe.py::test_partitions ERROR                                                                                                          [ 73%]
distributed/pipeline/sync/test_pipe.py::test_merged_partitions SKIPPED                                                                                                 [ 76%]
distributed/pipeline/sync/test_pipe.py::test_deny_moving ERROR                                                                                                         [ 78%]
distributed/pipeline/sync/test_pipe.py::test_empty_module ERROR                                                                                                        [ 81%]
distributed/pipeline/sync/test_pipe.py::test_named_children ERROR                                                                                                      [ 84%]
distributed/pipeline/sync/test_pipe.py::test_verify_module_non_sequential ERROR                                                                                        [ 86%]
distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_children ERROR                                                                                    [ 89%]
distributed/pipeline/sync/test_pipe.py::test_verify_module_params_on_same_device SKIPPED                                                                               [ 92%]
distributed/pipeline/sync/test_pipe.py::test_verify_nested_modules SKIPPED                                                                                             [ 94%]
distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_parameters_on_same_device ERROR                                                                   [ 97%]
distributed/pipeline/sync/test_pipe.py::test_forward_lockstep ERROR                                                                                                    [100%]

=================================================================================== ERRORS ===================================================================================
_____________________________________________________________________ ERROR at setup of test_parameters ______________________________________________________________________
Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 237, in _init_rpc_backend
    rpc_agent = backend_registry.init_backend(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 99, in init_backend
    return backend.value.init_backend_handler(*args, **kwargs)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/backend_registry.py", line 313, in _tensorpipe_init_backend_handler
    api._init_rpc_states(agent)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/api.py", line 116, in _init_rpc_states
    _set_and_start_rpc_agent(agent)
RuntimeError: In operator() at tensorpipe/common/ibv.h:151 "": Cannot allocate memory
____________________________________________________________________ ERROR at setup of test_public_attrs _____________________________________________________________________
Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 234, in _init_rpc_backend
    raise RuntimeError("RPC is already initialized")
RuntimeError: RPC is already initialized
___________________________________________________________________ ERROR at setup of test_sequential_like ___________________________________________________________________
Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 234, in _init_rpc_backend
    raise RuntimeError("RPC is already initialized")
RuntimeError: RPC is already initialized
_________________________________________________________________ ERROR at setup of test_chunks_less_than_1 __________________________________________________________________
Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 234, in _init_rpc_backend
    raise RuntimeError("RPC is already initialized")
RuntimeError: RPC is already initialized
_______________________________________________________________ ERROR at setup of test_batch_size_indivisible ________________________________________________________________
Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 234, in _init_rpc_backend
    raise RuntimeError("RPC is already initialized")
RuntimeError: RPC is already initialized
__________________________________________________________________ ERROR at setup of test_batch_size_small ___________________________________________________________________
Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 234, in _init_rpc_backend
    raise RuntimeError("RPC is already initialized")
RuntimeError: RPC is already initialized
___________________________________________________________________ ERROR at setup of test_checkpoint_mode ___________________________________________________________________
Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 234, in _init_rpc_backend
    raise RuntimeError("RPC is already initialized")
RuntimeError: RPC is already initialized
_______________________________________________________________ ERROR at setup of test_checkpoint_mode_invalid _______________________________________________________________
Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 234, in _init_rpc_backend
    raise RuntimeError("RPC is already initialized")
RuntimeError: RPC is already initialized
____________________________________________________________ ERROR at setup of test_checkpoint_mode_when_chunks_1 ____________________________________________________________@Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 234, in _init_rpc_backend
    raise RuntimeError("RPC is already initialized")
RuntimeError: RPC is already initialized
___________________________________________________________________ ERROR at setup of test_checkpoint_eval ___________________________________________________________________
Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 234, in _init_rpc_backend
    raise RuntimeError("RPC is already initialized")
RuntimeError: RPC is already initialized
_____________________________________________________________ ERROR at setup of test_checkpoint_non_float_input ______________________________________________________________
Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 234, in _init_rpc_backend
    raise RuntimeError("RPC is already initialized")
RuntimeError: RPC is already initialized
_______________________________________________________________________ ERROR at setup of test_no_grad _______________________________________________________________________
Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 234, in _init_rpc_backend
    raise RuntimeError("RPC is already initialized")
RuntimeError: RPC is already initialized
______________________________________________________________________ ERROR at setup of test_exception ______________________________________________________________________
Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 234, in _init_rpc_backend
    raise RuntimeError("RPC is already initialized")
RuntimeError: RPC is already initialized
______________________________________________________________ ERROR at setup of test_exception_early_stop_asap ______________________________________________________________
Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 234, in _init_rpc_backend
    raise RuntimeError("RPC is already initialized")
RuntimeError: RPC is already initialized
____________________________________________________________________ ERROR at setup of test_nested_input _____________________________________________________________________
Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 234, in _init_rpc_backend
    raise RuntimeError("RPC is already initialized")
RuntimeError: RPC is already initialized
_____________________________________________________________________ ERROR at setup of test_input_pair ______________________________________________________________________
Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 234, in _init_rpc_backend
    raise RuntimeError("RPC is already initialized")
RuntimeError: RPC is already initialized
___________________________________________________________________ ERROR at setup of test_input_singleton ___________________________________________________________________
Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 234, in _init_rpc_backend
    raise RuntimeError("RPC is already initialized")
RuntimeError: RPC is already initialized
____________________________________________________________________ ERROR at setup of test_input_varargs ____________________________________________________________________
Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 234, in _init_rpc_backend
    raise RuntimeError("RPC is already initialized")
RuntimeError: RPC is already initialized
_____________________________________________________________________ ERROR at setup of test_non_tensor ______________________________________________________________________
Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 234, in _init_rpc_backend
    raise RuntimeError("RPC is already initialized")
RuntimeError: RPC is already initialized
_________________________________________________________________ ERROR at setup of test_non_tensor_sequence _________________________________________________________________
Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 234, in _init_rpc_backend
    raise RuntimeError("RPC is already initialized")
RuntimeError: RPC is already initialized
_____________________________________________________________ ERROR at setup of test_deferred_batch_norm[never] ______________________________________________________________
Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 234, in _init_rpc_backend
    raise RuntimeError("RPC is already initialized")
RuntimeError: RPC is already initialized
_____________________________________________________________ ERROR at setup of test_deferred_batch_norm[always] _____________________________________________________________
Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 234, in _init_rpc_backend
    raise RuntimeError("RPC is already initialized")
RuntimeError: RPC is already initialized
__________________________________________________________ ERROR at setup of test_deferred_batch_norm[except_last] ___________________________________________________________
Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 234, in _init_rpc_backend
    raise RuntimeError("RPC is already initialized")
RuntimeError: RPC is already initialized
__________________________________________________________ ERROR at setup of test_deferred_batch_norm_params[never] __________________________________________________________
Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 234, in _init_rpc_backend
    raise RuntimeError("RPC is already initialized")
RuntimeError: RPC is already initialized
_________________________________________________________ ERROR at setup of test_deferred_batch_norm_params[always] __________________________________________________________
Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 234, in _init_rpc_backend
    raise RuntimeError("RPC is already initialized")
RuntimeError: RPC is already initialized
_______________________________________________________________________ ERROR at setup of test_devices _______________________________________________________________________
Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 234, in _init_rpc_backend
    raise RuntimeError("RPC is already initialized")
RuntimeError: RPC is already initialized
_____________________________________________________________________ ERROR at setup of test_partitions ______________________________________________________________________
Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 234, in _init_rpc_backend
    raise RuntimeError("RPC is already initialized")
RuntimeError: RPC is already initialized
_____________________________________________________________________ ERROR at setup of test_deny_moving _____________________________________________________________________
Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 234, in _init_rpc_backend
    raise RuntimeError("RPC is already initialized")
RuntimeError: RPC is already initialized
____________________________________________________________________ ERROR at setup of test_empty_module _____________________________________________________________________
Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 234, in _init_rpc_backend
    raise RuntimeError("RPC is already initialized")
RuntimeError: RPC is already initialized
___________________________________________________________________ ERROR at setup of test_named_children ____________________________________________________________________
Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 234, in _init_rpc_backend
    raise RuntimeError("RPC is already initialized")
RuntimeError: RPC is already initialized
____________________________________________________________ ERROR at setup of test_verify_module_non_sequential _____________________________________________________________
Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 234, in _init_rpc_backend
    raise RuntimeError("RPC is already initialized")
RuntimeError: RPC is already initialized
__________________________________________________________ ERROR at setup of test_verify_module_duplicate_children ___________________________________________________________
Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 234, in _init_rpc_backend
    raise RuntimeError("RPC is already initialized")
RuntimeError: RPC is already initialized
__________________________________________________ ERROR at setup of test_verify_module_duplicate_parameters_on_same_device __________________________________________________
Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 234, in _init_rpc_backend
    raise RuntimeError("RPC is already initialized")
RuntimeError: RPC is already initialized
__________________________________________________________________ ERROR at setup of test_forward_lockstep ___________________________________________________________________
Traceback (most recent call last):
  File "/tmp/easybuild/build/PyTorch/1.9.0/foss-2020b/pytorch/test/distributed/pipeline/sync/conftest.py", line 44, in setup_rpc
    dist.rpc.init_rpc(
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 203, in init_rpc
    _init_rpc_backend(backend, store, name, rank, world_size, rpc_backend_options)
  File "/tmp/eb-6e3wfvku/tmpfm41gft2/lib/python3.8/site-packages/torch/distributed/rpc/__init__.py", line 234, in _init_rpc_backend
    raise RuntimeError("RPC is already initialized")
RuntimeError: RPC is already initialized
========================================================================== short test summary info ===========================================================================
SKIPPED [1] distributed/pipeline/sync/test_pipe.py:477: cuda required
SKIPPED [1] distributed/pipeline/sync/test_pipe.py:576: cuda required
SKIPPED [1] distributed/pipeline/sync/test_pipe.py:594: Need atleast two GPUs
ERROR distributed/pipeline/sync/test_pipe.py::test_parameters - RuntimeError: In operator() at tensorpipe/common/ibv.h:151 "": Cannot allocate memory
ERROR distributed/pipeline/sync/test_pipe.py::test_public_attrs - RuntimeError: RPC is already initialized
ERROR distributed/pipeline/sync/test_pipe.py::test_sequential_like - RuntimeError: RPC is already initialized
ERROR distributed/pipeline/sync/test_pipe.py::test_chunks_less_than_1 - RuntimeError: RPC is already initialized
ERROR distributed/pipeline/sync/test_pipe.py::test_batch_size_indivisible - RuntimeError: RPC is already initialized
ERROR distributed/pipeline/sync/test_pipe.py::test_batch_size_small - RuntimeError: RPC is already initialized
ERROR distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode - RuntimeError: RPC is already initialized
ERROR distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_invalid - RuntimeError: RPC is already initialized
ERROR distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_when_chunks_1 - RuntimeError: RPC is already initialized
ERROR distributed/pipeline/sync/test_pipe.py::test_checkpoint_eval - RuntimeError: RPC is already initialized
ERROR distributed/pipeline/sync/test_pipe.py::test_checkpoint_non_float_input - RuntimeError: RPC is already initialized
ERROR distributed/pipeline/sync/test_pipe.py::test_no_grad - RuntimeError: RPC is already initialized
ERROR distributed/pipeline/sync/test_pipe.py::test_exception - RuntimeError: RPC is already initialized
ERROR distributed/pipeline/sync/test_pipe.py::test_exception_early_stop_asap - RuntimeError: RPC is already initialized
ERROR distributed/pipeline/sync/test_pipe.py::test_nested_input - RuntimeError: RPC is already initialized
ERROR distributed/pipeline/sync/test_pipe.py::test_input_pair - RuntimeError: RPC is already initialized
ERROR distributed/pipeline/sync/test_pipe.py::test_input_singleton - RuntimeError: RPC is already initialized
ERROR distributed/pipeline/sync/test_pipe.py::test_input_varargs - RuntimeError: RPC is already initialized
ERROR distributed/pipeline/sync/test_pipe.py::test_non_tensor - RuntimeError: RPC is already initialized
ERROR distributed/pipeline/sync/test_pipe.py::test_non_tensor_sequence - RuntimeError: RPC is already initialized
ERROR distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[never] - RuntimeError: RPC is already initialized
ERROR distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[always] - RuntimeError: RPC is already initialized
ERROR distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[except_last] - RuntimeError: RPC is already initialized
ERROR distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[never] - RuntimeError: RPC is already initialized
ERROR distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[always] - RuntimeError: RPC is already initialized
ERROR distributed/pipeline/sync/test_pipe.py::test_devices - RuntimeError: RPC is already initialized
ERROR distributed/pipeline/sync/test_pipe.py::test_partitions - RuntimeError: RPC is already initialized
ERROR distributed/pipeline/sync/test_pipe.py::test_deny_moving - RuntimeError: RPC is already initialized
ERROR distributed/pipeline/sync/test_pipe.py::test_empty_module - RuntimeError: RPC is already initialized
ERROR distributed/pipeline/sync/test_pipe.py::test_named_children - RuntimeError: RPC is already initialized
ERROR distributed/pipeline/sync/test_pipe.py::test_verify_module_non_sequential - RuntimeError: RPC is already initialized
ERROR distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_children - RuntimeError: RPC is already initialized
ERROR distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_parameters_on_same_device - RuntimeError: RPC is already initialized
ERROR distributed/pipeline/sync/test_pipe.py::test_forward_lockstep - RuntimeError: RPC is already initialized
================================================================== 1 passed, 3 skipped, 34 errors in 0.25s ===================================================================
distributed/pipeline/sync/test_pipe failed!
````

## Expected behavior

A non-failing build without test errors.

## Environment

OS: Red Hat Enterprise Linux release 8.4 (Ootpa) (x86_64)
GCC version: (GCC) 10.2.0
CMake version: version 3.18.4
Libc version: glibc-2.28

Python version: 3.8 (64-bit runtime)
Python platform: Linux-4.18.0-193.1.2.el8_2.x86_64-x86_64-with-glibc2.2.5
CUDA runtime version: 11.1.1
GPU models and configuration: Tesla T4
Nvidia driver version: 465.19.01

Versions of relevant libraries:
[pip3] numpy==1.19.4



cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @agolynski @SciPioneer @H-Huang @mrzzd @cbalioglu @gcramer23