shrinath-suresh/blogs-docs-splitted huggingface dataset:
default value set during initialization or with
        "_set_rpc_timeout" is used.
Returns:
      Returns a "Future" object that can be waited on. When completed,
      the return value of "func" on "args" and "kwargs" can be
      retrieved from the "Future" object.
Warning:
 Using GPU tensors as arguments or return values of "func" is not
 supported since we don't support sending GPU tensors over the
 wire. You need to explicitly copy GPU tensors to CPU before using
 them as arguments or return values of "func".

Warning:
 The "rpc_async" API does not copy storages of argument tensors
 until sending them over the wire, which could be done by a
 different thread depending on the RPC backend type. The caller
 should make sure that the contents of those tensors stay intact
 until the returned "Future" completes.

Example::
      Make sure that "MASTER_ADDR" and "MASTER_PORT" are set properly
on both workers. Refer to "init_process_group()" API for more
      details. For example,
  export MASTER_ADDR=localhost export MASTER_PORT=5678

  Then run the following code in two different processes:

  >>> # On worker 0:
  >>> import torch
  >>> import torch.distributed.rpc as rpc
  >>> rpc.init_rpc("worker0", rank=0, world_size=2)
  >>> fut1 = rpc.rpc_async("worker1", torch.add, args=(torch.ones(2), 3))
  >>> fut2 = rpc.rpc_async("worker1", min, args=(1, 2))
  >>> result = fut1.wait() + fut2.wait()
  >>> rpc.shutdown()

  >>> # On worker 1:
  >>> import torch.distributed.rpc as rpc
  >>> rpc.init_rpc("worker1", rank=1, world_size=2)
  >>> rpc.shutdown()

  Below is an example of running a TorchScript function using RPC.

  >>> # On both workers:
  >>> @torch.jit.script
  >>> def my_script_add(t1, t2):
  >>>    return torch.add(t1, t2)

  >>> # On worker 0:
  >>> import torch.distributed.rpc as rpc
import torch.distributed.rpc as rpc
      >>> rpc.init_rpc("worker0", rank=0, world_size=2)
      >>> fut = rpc.rpc_async("worker1", my_script_add, args=(torch.ones(2), 3))
      >>> ret = fut.wait()
      >>> rpc.shutdown()



  >>> # On worker 1:
  >>> import torch.distributed.rpc as rpc
  >>> rpc.init_rpc("worker1", rank=1, world_size=2)
  >>> rpc.shutdown()

torch.distributed.rpc.remote(to, func, args=None, kwargs=None, timeout=- 1.0)
Make a remote call to run "func" on worker "to" and return an
   "RRef" to the result value immediately. Worker "to" will be the
   owner of the returned "RRef", and the worker calling "remote" is a
   user. The owner manages the global reference count of its "RRef",
   and the owner "RRef" is only destructed when globally there are no
   living references to it.
Parameters:
      * to (str or WorkerInfo or int) --
        name/rank/"WorkerInfo" of the destination worker.
func (Callable) -- a callable function, such as Python
        callables, builtin operators (e.g. "add()") and annotated
        TorchScript functions.


args (tuple) -- the argument tuple for the "func"
    invocation.


kwargs (dict) -- is a dictionary of keyword arguments
    for the "func" invocation.


timeout (float, optional) -- timeout in seconds for
    this remote call. If the creation of this "RRef" on worker
    "to" is not successfully processed on this worker within this
    timeout, then the next time there is an attempt to use the
    RRef (such as "to_here()"), a timeout will be raised
    indicating this failure. A value of 0 indicates an infinite
    timeout, i.e. a timeout error will never be raised. If not
    provided, the default value set during initialization or with
    "_set_rpc_timeout" is used.




Returns:
      A user "RRef" instance to the result value. Use the blocking API
"torch.distributed.rpc.RRef.to_here()" to retrieve the result
      value locally.
Warning:
 The "remote" API does not copy storages of argument tensors until
 sending them over the wire, which could be done by a different
 thread depending on the RPC backend type. The caller should make
 sure that the contents of those tensors stay intact until the
 returned RRef is confirmed by the owner, which can be checked
 using the "torch.distributed.rpc.RRef.confirmed_by_owner()" API.

Warning:
 Errors such as timeouts for the "remote" API are handled on a
 best-effort basis. This means that when remote calls initiated by
 "remote" fail, such as with a timeout error, we take a best-
 effort approach to error handling. This means that errors are
 handled and set on the resulting RRef on an asynchronous basis.
 If the RRef has not been used by the application before this
 handling (such as "to_here" or fork call), then future uses of
the "RRef" will appropriately raise errors. However, it is
     possible that the user application will use the "RRef" before the
     errors are handled. In this case, errors may not be raised as
     they have not yet been handled.
Example:
  Make sure that ``MASTER_ADDR`` and ``MASTER_PORT`` are set properly
  on both workers. Refer to :meth:`~torch.distributed.init_process_group`
  API for more details. For example,

  export MASTER_ADDR=localhost
  export MASTER_PORT=5678

  Then run the following code in two different processes:

  >>> # On worker 0:
  >>> import torch
  >>> import torch.distributed.rpc as rpc
  >>> rpc.init_rpc("worker0", rank=0, world_size=2)
  >>> rref1 = rpc.remote("worker1", torch.add, args=(torch.ones(2), 3))
  >>> rref2 = rpc.remote("worker1", torch.add, args=(torch.ones(2), 1))
  >>> x = rref1.to_here() + rref2.to_here()
  >>> rpc.shutdown()

  >>> # On worker 1:
  >>> import torch.distributed.rpc as rpc
import torch.distributed.rpc as rpc
      >>> rpc.init_rpc("worker1", rank=1, world_size=2)
      >>> rpc.shutdown()



  Below is an example of running a TorchScript function using RPC.

  >>> # On both workers:
  >>> @torch.jit.script
  >>> def my_script_add(t1, t2):
  >>>    return torch.add(t1, t2)

  >>> # On worker 0:
  >>> import torch.distributed.rpc as rpc
  >>> rpc.init_rpc("worker0", rank=0, world_size=2)
  >>> rref = rpc.remote("worker1", my_script_add, args=(torch.ones(2), 3))
  >>> rref.to_here()
  >>> rpc.shutdown()

  >>> # On worker 1:
  >>> import torch.distributed.rpc as rpc
  >>> rpc.init_rpc("worker1", rank=1, world_size=2)
  >>> rpc.shutdown()

torch.distributed.rpc.get_worker_info(worker_name=None)
Get "WorkerInfo" of a given worker name. Use this "WorkerInfo" to
   avoid passing an expensive string on every invocation.
Parameters:
      worker_name (str) -- the string name of a worker. If
"None", return the the id of the current worker. (default
      "None")
Returns:
      "WorkerInfo" instance for the given "worker_name" or
      "WorkerInfo" of the current worker if "worker_name" is "None".
torch.distributed.rpc.shutdown(graceful=True, timeout=0)
Perform a shutdown of the RPC agent, and then destroy the RPC
   agent. This stops the local agent from accepting outstanding
   requests, and shuts down the RPC framework by terminating all RPC
   threads. If "graceful=True", this will block until all local and
   remote RPC processes reach this method and wait for all outstanding
   work to complete. Otherwise, if "graceful=False", this is a local
   shutdown, and it does not wait for other RPC processes to reach
   this method.
Warning:
 For "Future" objects returned by "rpc_async()", "future.wait()"
 should not be called after "shutdown()".

Parameters:
      graceful (bool) -- Whether to do a graceful shutdown or
not. If True, this will 1) wait until there is no pending system
      messages for "UserRRefs" and delete them; 2) block until all
      local and remote RPC processes have reached this method and wait
      for all outstanding work to complete.
Example::
      Make sure that "MASTER_ADDR" and "MASTER_PORT" are set properly
      on both workers. Refer to "init_process_group()" API for more
      details. For example,
  export MASTER_ADDR=localhost export MASTER_PORT=5678

  Then run the following code in two different processes:

  >>> # On worker 0:
  >>> import torch
  >>> import torch.distributed.rpc as rpc
  >>> rpc.init_rpc("worker0", rank=0, world_size=2)
  >>> # do some work
  >>> result = rpc.rpc_sync("worker1", torch.add, args=(torch.ones(1), 1))
  >>> # ready to shutdown
  >>> rpc.shutdown()

  >>> # On worker 1:
  >>> import torch.distributed.rpc as rpc
  >>> rpc.init_rpc("worker1", rank=1, world_size=2)
wait for worker 0 to finish work, and then shutdown.
  >>> rpc.shutdown()




class torch.distributed.rpc.WorkerInfo
A structure that encapsulates information of a worker in the
   system. Contains the name and ID of the worker. This class is not
   meant to be constructed directly, rather, an instance can be
   retrieved through "get_worker_info()" and the result can be passed
   in to functions such as "rpc_sync()", "rpc_async()", "remote()" to
   avoid copying a string on every invocation.
property id
  Globally unique id to identify the worker.

property name
  The name of the worker.

The RPC package also provides decorators which allow applications to
specify how a given function should be treated on the callee side.
torch.distributed.rpc.functions.async_execution(fn)
A decorator for a function indicating that the return value of the
   function is guaranteed to be a "Future" object and this function
can run asynchronously on the RPC callee. More specifically, the
   callee extracts the "Future" returned by the wrapped function and
   installs subsequent processing steps as a callback to that
   "Future". The installed callback will read the value from the
   "Future" when completed and send the value back as the RPC
   response. That also means the returned "Future" only exists on the
   callee side and is never sent through RPC. This decorator is useful
   when the wrapped function's ("fn") execution needs to pause and
   resume due to, e.g., containing "rpc_async()" or waiting for other
   signals.
Note:
 To enable asynchronous execution, applications must pass the
 function object returned by this decorator to RPC APIs. If RPC
 detected attributes installed by this decorator, it knows that
 this function returns a "Future" object and will handle that
 accordingly. However, this does not mean this decorator has to be
outmost one when defining a function. For example, when combined
     with "@staticmethod" or "@classmethod",
     "@rpc.functions.async_execution" needs to be the inner decorator
     to allow the target function be recognized as a static or class
     function. This target function can still execute asynchronously
     because, when accessed, the static or class method preserves
     attributes installed by "@rpc.functions.async_execution".
Example::
      The returned "Future" object can come from "rpc_async()",
      "then()", or "Future" constructor. The example below shows
      directly using the "Future" returned by "then()".
  >>> from torch.distributed import rpc
  >>>
  >>> # omitting setup and shutdown RPC
  >>>
  >>> # On all workers
  >>> @rpc.functions.async_execution
  >>> def async_add_chained(to, x, y, z):
  >>>     # This function runs on "worker1" and returns immediately when
# the callback is installed through the `then(cb)` API. In the
  >>>     # mean time, the `rpc_async` to "worker2" can run concurrently.
  >>>     # When the return value of that `rpc_async` arrives at
  >>>     # "worker1", "worker1" will run the lambda function accordingly
  >>>     # and set the value for the previously returned `Future`, which
  >>>     # will then trigger RPC to send the result back to "worker0".
  >>>     return rpc.rpc_async(to, torch.add, args=(x, y)).then(
  >>>         lambda fut: fut.wait() + z
  >>>     )
  >>>
  >>> # On worker0
  >>> ret = rpc.rpc_sync(
  >>>     "worker1",
  >>>     async_add_chained,
  >>>     args=("worker2", torch.ones(2), 1, 1)
  >>> )
  >>> print(ret)  # prints tensor([3., 3.])




  When combined with TorchScript decorators, this decorator must
  be the outmost one.

  >>> from torch import Tensor
  >>> from torch.futures import Future
from torch.futures import Future
      >>> from torch.distributed import rpc
      >>>
      >>> # omitting setup and shutdown RPC
      >>>
      >>> # On all workers
      >>> @torch.jit.script
      >>> def script_add(x: Tensor, y: Tensor) -> Tensor:
      >>>     return x + y
      >>>
      >>> @rpc.functions.async_execution
      >>> @torch.jit.script
      >>> def async_add(to: str, x: Tensor, y: Tensor) -> Future[Tensor]:
      >>>     return rpc.rpc_async(to, script_add, (x, y))
      >>>
      >>> # On worker0
      >>> ret = rpc.rpc_sync(
      >>>     "worker1",
      >>>     async_add,
      >>>     args=("worker2", torch.ones(2), 1)
      >>> )
      >>> print(ret)  # prints tensor([2., 2.])



  When combined with static or class method, this decorator must
  be the inner one.

  >>> from torch.distributed import rpc
  >>>
  >>> # omitting setup and shutdown RPC
  >>>
  >>> # On all workers
  >>> class AsyncExecutionClass:
  >>>
class AsyncExecutionClass:
      >>>
      >>>     @staticmethod
      >>>     @rpc.functions.async_execution
      >>>     def static_async_add(to, x, y, z):
      >>>         return rpc.rpc_async(to, torch.add, args=(x, y)).then(
      >>>             lambda fut: fut.wait() + z
      >>>         )
      >>>
      >>>     @classmethod
      >>>     @rpc.functions.async_execution
      >>>     def class_async_add(cls, to, x, y, z):
      >>>         ret_fut = torch.futures.Future()
      >>>         rpc.rpc_async(to, torch.add, args=(x, y)).then(
      >>>             lambda fut: ret_fut.set_result(fut.wait() + z)
      >>>         )
      >>>         return ret_fut
      >>>
      >>>     @rpc.functions.async_execution
      >>>     def bound_async_add(self, to, x, y, z):
      >>>         return rpc.rpc_async(to, torch.add, args=(x, y)).then(
      >>>             lambda fut: fut.wait() + z
      >>>         )
      >>>
      >>> # On worker0
      >>> ret = rpc.rpc_sync(
      >>>     "worker1",
"worker1",
  >>>     AsyncExecutionClass.static_async_add,
  >>>     args=("worker2", torch.ones(2), 1, 2)
  >>> )
  >>> print(ret)  # prints tensor([4., 4.])
  >>>
  >>> ret = rpc.rpc_sync(
  >>>     "worker1",
  >>>     AsyncExecutionClass.class_async_add,
  >>>     args=("worker2", torch.ones(2), 1, 2)
  >>> )
  >>> print(ret)  # prints tensor([4., 4.])




  This decorator also works with RRef helpers, i.e., .
  "torch.distributed.rpc.RRef.rpc_sync()",
  "torch.distributed.rpc.RRef.rpc_async()", and
  "torch.distributed.rpc.RRef.remote()".

  >>> from torch.distributed import rpc
  >>>
  >>> # reuse the AsyncExecutionClass class above
  >>> rref = rpc.remote("worker1", AsyncExecutionClass)
  >>> ret = rref.rpc_sync().static_async_add("worker2", torch.ones(2), 1, 2)
  >>> print(ret)  # prints tensor([4., 4.])
  >>>
  >>> rref = rpc.remote("worker1", AsyncExecutionClass)
ret = rref.rpc_async().static_async_add("worker2", torch.ones(2), 1, 2).wait()
      >>> print(ret)  # prints tensor([4., 4.])
      >>>
      >>> rref = rpc.remote("worker1", AsyncExecutionClass)
      >>> ret = rref.remote().static_async_add("worker2", torch.ones(2), 1, 2).to_here()
      >>> print(ret)  # prints tensor([4., 4.])



Backends
The RPC module can leverage different backends to perform the
communication between the nodes. The backend to be used can be
specified in the "init_rpc()" function, by passing a certain value of
the "BackendType" enum. Regardless of what backend is used, the rest
of the RPC API won't change. Each backend also defines its own
subclass of the "RpcBackendOptions" class, an instance of which can
also be passed to "init_rpc()" to configure the backend's behavior.
class torch.distributed.rpc.BackendType(value)
An enum class of available backends.
PyTorch ships with a builtin "BackendType.TENSORPIPE" backend.
Additional ones can be registered using the "register_backend()"
   function.
class torch.distributed.rpc.RpcBackendOptions
An abstract structure encapsulating the options passed into the RPC
   backend. An instance of this class can be passed in to "init_rpc()"
   in order to initialize RPC with specific configurations, such as
   the RPC timeout and "init_method" to be used.
property init_method
  URL specifying how to initialize the process group. Default is
  "env://"

property rpc_timeout
  A float indicating the timeout to use for all RPCs. If an RPC
  does not complete in this timeframe, it will complete with an
  exception indicating that it has timed out.

TensorPipe Backend
~~~~~~~~~~~~~~~~~~
The TensorPipe agent, which is the default, leverages the TensorPipe
library, which provides a natively point-to-point communication
primitive specifically suited for machine learning that fundamentally
addresses some of the limitations of Gloo. Compared to Gloo, it has
the advantage of being asynchronous, which allows a large number of
transfers to occur simultaneously, each at their own speed, without
blocking each other. It will only open pipes between pairs of nodes
when needed, on demand, and when one node fails only its incident
pipes will be closed, while all other ones will keep working as
normal. In addition, it is able to support multiple different
transports (TCP, of course, but also shared memory, NVLink,
InfiniBand, ...) and can automatically detect their availability and
negotiate the best transport to use for each pipe.
The TensorPipe backend has been introduced in PyTorch v1.6 and is
being actively developed. At the moment, it only supports CPU tensors,
with GPU support coming soon. It comes with a TCP-based transport,
just like Gloo. It is also able to automatically chunk and multiplex
large tensors over multiple sockets and threads in order to achieve
very high bandwidths. The agent will be able to pick the best
transport on its own, with no intervention required.
Example:



import os
from torch.distributed import rpc
os.environ['MASTER_ADDR'] = 'localhost'
os.environ['MASTER_PORT'] = '29500'
rpc.init_rpc(
    "worker1",
    rank=0,
    world_size=2,
    rpc_backend_options=rpc.TensorPipeRpcBackendOptions(
        num_worker_threads=8,
        rpc_timeout=20 # 20 second timeout
    )
)
omitting init_rpc invocation on worker2



class torch.distributed.rpc.TensorPipeRpcBackendOptions(*, num_worker_threads=16, rpc_timeout=60.0, init_method='env://', device_maps=None, devices=None, _transports=None, _channels=None)
The backend options for "TensorPipeAgent", derived from
   "RpcBackendOptions".
Parameters:
      * num_worker_threads (int, optional) -- The number of
        threads in the thread-pool used by "TensorPipeAgent" to
        execute requests (default: 16).
execute requests (default: 16).
  * **rpc_timeout** (*float**, **optional*) -- The default
    timeout, in seconds, for RPC requests (default: 60 seconds).
    If the RPC has not completed in this timeframe, an exception
    indicating so will be raised. Callers can override this
    timeout for individual RPCs in "rpc_sync()" and "rpc_async()"
    if necessary.

  * **init_method** (*str**, **optional*) -- The URL to initialize
    the distributed store used for rendezvous. It takes any value
    accepted for the same argument of "init_process_group()"
    (default: "env://").

  * **device_maps** (*Dict**[**str**, **Dict**]**, **optional*) --
    Device placement mappings from this worker to the callee. Key
    is the callee worker name and value the dictionary ("Dict" of
    "int", "str", or "torch.device") that maps this worker's
    devices to the callee worker's devices. (default: "None")
devices (List[int, str, or "torch.device"], optional) --
        all local CUDA devices used by RPC agent. By Default, it will
        be initialized to all local devices from its own "device_maps"
        and corresponding devices from its peers' "device_maps". When
        processing CUDA RPC requests, the agent will properly
        synchronize CUDA streams for all devices in this "List".

property device_maps
  The device map locations.

property devices
  All devices used by the local agent.

property init_method
  URL specifying how to initialize the process group. Default is
  "env://"

property num_worker_threads
  The number of threads in the thread-pool used by
  "TensorPipeAgent" to execute requests.

property rpc_timeout
  A float indicating the timeout to use for all RPCs. If an RPC
  does not complete in this timeframe, it will complete with an
  exception indicating that it has timed out.

set_device_map(to, device_map)
set_device_map(to, device_map)
  Set device mapping between each RPC caller and callee pair. This
  function can be called multiple times to incrementally add
  device placement configurations.

  Parameters:
     * **to** (*str*) -- Callee name.

     * **device_map** (*Dict of python:int**, **str**, or
       **torch.device*) -- Device placement mappings from this
       worker to the callee. This map must be invertible.

  -[ Example ]-

  >>> # both workers
  >>> def add(x, y):
  >>>     print(x)  # tensor([1., 1.], device='cuda:1')
  >>>     return x + y, (x + y).to(2)
  >>>
  >>> # on worker 0
  >>> options = TensorPipeRpcBackendOptions(
  >>>     num_worker_threads=8,
  >>>     device_maps={"worker1": {0: 1}}
  >>>     # maps worker0's cuda:0 to worker1's cuda:1
  >>> )
  >>> options.set_device_map("worker1", {1: 2})
  >>> # maps worker0's cuda:1 to worker1's cuda:2
  >>>
  >>> rpc.init_rpc(
>>> rpc.init_rpc(
  >>>     "worker0",
  >>>     rank=0,
  >>>     world_size=2,
  >>>     backend=rpc.BackendType.TENSORPIPE,
  >>>     rpc_backend_options=options
  >>> )
  >>>
  >>> x = torch.ones(2)
  >>> rets = rpc.rpc_sync("worker1", add, args=(x.to(0), 1))
  >>> # The first argument will be moved to cuda:1 on worker1. When
  >>> # sending the return value back, it will follow the invert of
  >>> # the device map, and hence will be moved back to cuda:0 and
  >>> # cuda:1 on worker0
  >>> print(rets[0])  # tensor([2., 2.], device='cuda:0')
  >>> print(rets[1])  # tensor([2., 2.], device='cuda:1')




set_devices(devices)
  Set local devices used by the TensorPipe RPC agent. When
  processing CUDA RPC requests, the TensorPipe RPC agent will
  properly synchronize CUDA streams for all devices in this
  "List".

  Parameters:
     **devices** (*List of python:int**, **str**, or
*torch.device) -- local devices used by the TensorPipe RPC
         agent.
Note:
The RPC framework does not automatically retry any "rpc_sync()",
  "rpc_async()" and "remote()" calls. The reason being that there is
  no way the RPC framework can determine whether an operation is
  idempotent or not and whether it is safe to retry. As a result, it
  is the application's responsibility to deal with failures and retry
  if necessary. RPC communication is based on TCP and as a result
  failures could happen due to network failures or intermittent
  network connectivity issues. In such scenarios, the application
  needs to retry appropriately with reasonable backoffs to ensure the
  network isn't overwhelmed by aggressive retries.
RRef
Warning:
RRefs are not currently supported when using CUDA tensors
An "RRef" (Remote REFerence) is a reference to a value of some type
"T" (e.g. "Tensor") on a remote worker. This handle keeps the
referenced remote value alive on the owner, but there is no
implication that the value will be transferred to the local worker in
the future. RRefs can be used in multi-machine training by holding
references to nn.Modules that exist on other workers, and calling the
appropriate functions to retrieve or modify their parameters during
training. See Remote Reference Protocol for more details.
class torch.distributed.rpc.RRef
More Information about RRef
^^^^^^^^^^^^^^^^^^^^^^^^^^^


Remote Reference Protocol


Background


Assumptions


RRef Lifetime


Design Reasoning


Implementation




Protocol Scenarios


User Share RRef with Owner as Return Value


User Share RRef with Owner as Argument


Owner Share RRef with User


User Share RRef with User




RemoteModule
Warning:
RemoteModule is not currently supported when using CUDA tensors
"RemoteModule" is an easy way to create an nn.Module remotely on a
different process. The actual module resides on a remote host, but the