shrinath-suresh/blogs-docs-splitted huggingface dataset:
fut = torch.futures.Future()
         >>> # The inserted callback will print the return value when
         >>> # receiving the response from "worker1"
         >>> cb_fut = fut.then(callback)
         >>> chain_cb_fut = cb_fut.then(
         ...     lambda x : print(f"Chained cb done. {x.wait()}")
         ... )
         >>> fut.set_result(5)
         RPC return value is 5.
         Chained cb done. None



value()
  Obtain the value of an already-completed future.

  This method should only be called after a call to "wait()" has
  completed, or inside a callback function passed to "then()". In
  other cases this "Future" may not yet hold a value and calling
  "value()" could fail.

  If the value contains tensors that reside on GPUs, then this
  method will *not* perform any additional synchronization. This
  should be done beforehand, separately, through a call to
  "wait()" (except within callbacks, for which it's already being
taken care of by "then()").
  Returns:
     The value held by this "Future". If the function (callback or
     RPC) creating the value has thrown an error, this "value()"
     method will also throw an error.

  Return type:
     *T*

wait()
  Block until the value of this "Future" is ready.

  If the value contains tensors that reside on GPUs, then an
  additional synchronization is performed with the kernels
  (executing on the device) which may be asynchronously populating
  those tensors. Such sync is non-blocking, which means that
  "wait()" will insert the necessary instructions in the current
  streams to ensure that further operations enqueued on those
  streams will be properly scheduled after the async kernels but,
  once that is done, "wait()" will return, even if those kernels
  are still running. No further synchronization is required when
  accessing and using the values, as long as one doesn't change
  streams.
streams.
  Returns:
     The value held by this "Future". If the function (callback or
     RPC) creating the value has thrown an error, this "wait"
     method will also throw an error.

  Return type:
     *T*

torch.futures.collect_all(futures)
Collects the provided "Future" objects into a single combined
   "Future" that is completed when all of the sub-futures are
   completed.
Parameters:
      futures (list) -- a list of "Future" objects.
Returns:
      Returns a "Future" object to a list of the passed in Futures.
Return type:
      Future[List[Future]]
Example::
      >>> fut0 = torch.futures.Future()
      >>> fut1 = torch.futures.Future()
      >>> fut = torch.futures.collect_all([fut0, fut1])
      >>> fut0.set_result(0)
      >>> fut1.set_result(1)
      >>> fut_list = fut.wait()
      >>> print(f"fut0 result = {fut_list[0].wait()}")
      fut0 result = 0
      >>> print(f"fut1 result = {fut_list[1].wait()}")
      fut1 result = 1
fut1 result = 1
torch.futures.wait_all(futures)
Waits for all provided futures to be complete, and returns the list
   of completed values. If any of the futures encounters an error, the
   method will exit early and report the error not waiting for other
   futures to complete.
Parameters:
      futures (list) -- a list of "Future" object.
Returns:
      A list of the completed "Future" results. This method will throw
      an error if "wait" on any "Future" throws.
Return type:
      List
torch.profiler
Overview
PyTorch Profiler is a tool that allows the collection of performance
metrics during training and inference. Profiler's context manager API
can be used to better understand what model operators are the most
expensive, examine their input shapes and stack traces, study device
kernel activity and visualize the execution trace.
Note:
An earlier version of the API in "torch.autograd" module is
  considered legacy and will be deprecated.
API Reference
class torch.profiler._KinetoProfile(*, activities=None, record_shapes=False, profile_memory=False, with_stack=False, with_flops=False, with_modules=False, experimental_config=None)
Low-level profiler wrap the autograd profile
Parameters:
      * activities (iterable) -- list of activity groups (CPU,
        CUDA) to use in profiling, supported values:
        "torch.profiler.ProfilerActivity.CPU",
        "torch.profiler.ProfilerActivity.CUDA". Default value:
ProfilerActivity.CPU and (when available)
        ProfilerActivity.CUDA.
  * **record_shapes** (*bool*) -- save information about
    operator's input shapes.

  * **profile_memory** (*bool*) -- track tensor memory
    allocation/deallocation.

  * **with_stack** (*bool*) -- record source information (file and
    line number) for the ops.

  * **with_flops** (*bool*) -- use formula to estimate the FLOPS
    of specific operators (matrix multiplication and 2D
    convolution).

  * **with_modules** (*bool*) -- record module hierarchy
    (including function names) corresponding to the callstack of
    the op. e.g. If module A's forward call's module B's forward
    which contains an aten::add op, then aten::add's module
    hierarchy is A.B Note that this support exist, at the moment,
    only for TorchScript models and not eager mode models.

  * **experimental_config** (*_ExperimentalConfig*) -- A set of
experimental options used by profiler libraries like Kineto.
        Note, backward compatibility is not guaranteed.
Note:
 This API is experimental and subject to change in the
 future.Enabling shape and stack tracing results in additional
 overhead. When record_shapes=True is specified, profiler will
 temporarily hold references to the tensors; that may further
 prevent certain optimizations that depend on the reference count
 and introduce extra tensor copies.

add_metadata(key, value)
  Adds a user defined metadata with a string key and a string
  value into the trace file

add_metadata_json(key, value)
  Adds a user defined metadata with a string key and a valid json
  value into the trace file

events()
  Returns the list of unaggregated profiler events, to be used in
  the trace callback or after the profiling is finished

export_chrome_trace(path)
  Exports the collected trace in Chrome JSON format.
export_stacks(path, metric='self_cpu_time_total')
  Save stack traces in a file in a format suitable for
  visualization.

  Parameters:
     * **path** (*str*) -- save stacks file to this location;

     * **metric** (*str*) -- metric to use: "self_cpu_time_total"
       or "self_cuda_time_total"

  Note:

    Example of using FlameGraph tool:

    * git clone https://github.com/brendangregg/FlameGraph

    * cd FlameGraph

    * ./flamegraph.pl --title "CPU time" --countname "us."
      profiler.stacks > perf_viz.svg

key_averages(group_by_input_shape=False, group_by_stack_n=0)
  Averages events, grouping them by operator name and (optionally)
  input shapes and stack.

  Note:

    To use shape/stack functionality make sure to set
    record_shapes/with_stack when creating profiler context
    manager.
manager.
class torch.profiler.profile(*, activities=None, schedule=None, on_trace_ready=None, record_shapes=False, profile_memory=False, with_stack=False, with_flops=False, with_modules=False, experimental_config=None, use_cuda=None)
Profiler context manager.
Parameters:
      * activities (iterable) -- list of activity groups (CPU,
        CUDA) to use in profiling, supported values:
        "torch.profiler.ProfilerActivity.CPU",
        "torch.profiler.ProfilerActivity.CUDA". Default value:
        ProfilerActivity.CPU and (when available)
        ProfilerActivity.CUDA.
  * **schedule** (*Callable*) -- callable that takes step (int) as
    a single parameter and returns "ProfilerAction" value that
    specifies the profiler action to perform at each step.

  * **on_trace_ready** (*Callable*) -- callable that is called at
    each step when "schedule" returns
    "ProfilerAction.RECORD_AND_SAVE" during the profiling.
record_shapes (bool) -- save information about
        operator's input shapes.


profile_memory (bool) -- track tensor memory
    allocation/deallocation.


with_stack (bool) -- record source information (file and
    line number) for the ops.


with_flops (bool) -- use formula to estimate the FLOPs
    (floating point operations) of specific operators (matrix
    multiplication and 2D convolution).


with_modules (bool) -- record module hierarchy
    (including function names) corresponding to the callstack of
    the op. e.g. If module A's forward call's module B's forward
    which contains an aten::add op, then aten::add's module
    hierarchy is A.B Note that this support exist, at the moment,
    only for TorchScript models and not eager mode models.


experimental_config (_ExperimentalConfig) -- A set of
    experimental options used for Kineto library features. Note,
backward compatibility is not guaranteed.
  * **use_cuda** (*bool*) --

    Deprecated since version 1.8.1: use "activities" instead.

Note:
 Use "schedule()" to generate the callable schedule. Non-default
 schedules are useful when profiling long training jobs and allow
 the user to obtain multiple traces at the different iterations of
 the training process. The default schedule simply records all the
 events continuously for the duration of the context manager.

Note:
 Use "tensorboard_trace_handler()" to generate result files for T
 ensorBoard:"on_trace_ready=torch.profiler.tensorboard_trace_hand
 ler(dir_name)"After profiling, result files can be found in the
 specified directory. Use the command:"tensorboard --logdir
 dir_name"to see the results in TensorBoard. For more information,
 see PyTorch Profiler TensorBoard Plugin

Note:
 Enabling shape and stack tracing results in additional overhead.
When record_shapes=True is specified, profiler will temporarily
     hold references to the tensors; that may further prevent certain
     optimizations that depend on the reference count and introduce
     extra tensor copies.
Examples:
  with torch.profiler.profile(
      activities=[
          torch.profiler.ProfilerActivity.CPU,
          torch.profiler.ProfilerActivity.CUDA,
      ]
  ) as p:
      code_to_profile()
  print(p.key_averages().table(
      sort_by="self_cuda_time_total", row_limit=-1))

Using the profiler's "schedule", "on_trace_ready" and "step"
   functions:
  # Non-default profiler schedule allows user to turn profiler on and off
  # on different iterations of the training loop;
  # trace_handler is called every time a new trace becomes available
  def trace_handler(prof):
      print(prof.key_averages().table(
          sort_by="self_cuda_time_total", row_limit=-1))
prof.export_chrome_trace("/tmp/test_trace_" + str(prof.step_num) + ".json")
  with torch.profiler.profile(
      activities=[
          torch.profiler.ProfilerActivity.CPU,
          torch.profiler.ProfilerActivity.CUDA,
      ],

      # In this example with wait=1, warmup=1, active=2,
      # profiler will skip the first step/iteration,
      # start warming up on the second, record
      # the third and the forth iterations,
      # after which the trace will become available
      # and on_trace_ready (when set) is called;
      # the cycle repeats starting with the next step

      schedule=torch.profiler.schedule(
          wait=1,
          warmup=1,
          active=2),
      on_trace_ready=trace_handler
      # on_trace_ready=torch.profiler.tensorboard_trace_handler('./log')
      # used when outputting for tensorboard
      ) as p:
          for iter in range(N):
for iter in range(N):
                  code_iteration_to_profile(iter)
                  # send a signal to the profiler that the next iteration has started
                  p.step()
step()
  Signals the profiler that the next profiling step has started.

class torch.profiler.ProfilerAction(value)
Profiler actions that can be taken at the specified intervals
class torch.profiler.ProfilerActivity
Members:
CPU
CUDA
property name
torch.profiler.schedule(*, wait, warmup, active, repeat=0, skip_first=0)
Returns a callable that can be used as profiler "schedule"
   argument. The profiler will skip the first "skip_first" steps, then
   wait for "wait" steps, then do the warmup for the next "warmup"
   steps, then do the active recording for the next "active" steps and
   then repeat the cycle starting with "wait" steps. The optional
   number of cycles is specified with the "repeat" parameter, the zero
value means that the cycles will continue until the profiling is
   finished.
Return type:
      Callable
torch.profiler.tensorboard_trace_handler(dir_name, worker_name=None, use_gzip=False)
Outputs tracing files to directory of "dir_name", then that
   directory can be directly delivered to tensorboard as logdir.
   "worker_name" should be unique for each worker in distributed
   scenario, it will be set to '[hostname]_[pid]' by default.
Intel Instrumentation and Tracing Technology APIs
torch.profiler.itt.is_available()
Check if ITT feature is available or not
torch.profiler.itt.mark(msg)
Describe an instantaneous event that occurred at some point.
Parameters:
      msg (str) -- ASCII message to associate with the event.
torch.profiler.itt.range_push(msg)
Pushes a range onto a stack of nested range span.  Returns zero-
   based depth of the range that is started.
Parameters:
Distributed RPC Framework
The distributed RPC framework provides mechanisms for multi-machine
model training through a set of primitives to allow for remote
communication, and a higher-level API to automatically differentiate
models split across several machines.
Warning:
APIs in the RPC package are stable. There are multiple ongoing work
  items to improve performance and error handling, which will ship in
  future releases.
Warning:
CUDA support was introduced in PyTorch 1.9 and is still a beta
  feature. Not all features of the RPC package are yet compatible with
  CUDA support and thus their use is discouraged. These unsupported
  features include: RRefs, JIT compatibility, dist autograd and dist
  optimizer, and profiling. These shortcomings will be addressed in
  future releases.
Note:
Please refer to PyTorch Distributed Overview for a brief
  introduction to all features related to distributed training.
Basics
Basics
The distributed RPC framework makes it easy to run functions remotely,
supports referencing remote objects without copying the real data
around, and provides autograd and optimizer APIs to transparently run
backward and update parameters across RPC boundaries. These features
can be categorized into four sets of APIs.

Remote Procedure Call (RPC) supports running a function on the
   specified destination worker with the given arguments and getting
   the return value back or creating a reference to the return value.
   There are three main RPC APIs: "rpc_sync()" (synchronous),
   "rpc_async()" (asynchronous), and "remote()" (asynchronous and
   returns a reference to the remote return value). Use the
   synchronous API if the user code cannot proceed without the return
   value. Otherwise, use the asynchronous API to get a future, and
   wait on the future when the return value is needed on the caller.
   The "remote()" API is useful when the requirement is to create
something remotely but never need to fetch it to the caller.
   Imagine the case that a driver process is setting up a parameter
   server and a trainer. The driver can create an embedding table on
   the parameter server and then share the reference to the embedding
   table with the trainer, but itself will never use the embedding
   table locally. In this case, "rpc_sync()" and "rpc_async()" are no
   longer appropriate, as they always imply that the return value will
   be returned to the caller immediately or in the future.

Remote Reference (RRef) serves as a distributed shared pointer
   to a local or remote object. It can be shared with other workers
   and reference counting will be handled transparently. Each RRef
   only has one owner and the object only lives on that owner. Non-
   owner workers holding RRefs can get copies of the object from the
   owner by explicitly requesting it. This is useful when a worker
   needs to access some data object, but itself is neither the creator
(the caller of "remote()") or the owner of the object. The
   distributed optimizer, as we will discuss below, is one example of
   such use cases.


Distributed Autograd stitches together local autograd engines
   on all the workers involved in the forward pass, and automatically
   reach out to them during the backward pass to compute gradients.
   This is especially helpful if the forward pass needs to span
   multiple machines when conducting, e.g., distributed model parallel
   training, parameter-server training, etc. With this feature, user
   code no longer needs to worry about how to send gradients across
   RPC boundaries and in which order should the local autograd engines
   be launched, which can become quite complicated where there are
   nested and inter-dependent RPC calls in the forward pass.


Distributed Optimizer's constructor takes a "Optimizer()"
   (e.g., "SGD()", "Adagrad()", etc.) and a list of parameter RRefs,
creates an "Optimizer()" instance on each distinct RRef owner, and
   updates parameters accordingly when running "step()". When you have
   distributed forward and backward passes, parameters and gradients
   will be scattered across multiple workers, and hence it requires an
   optimizer on each of the involved workers. Distributed Optimizer
   wraps all those local optimizers into one, and provides a concise
   constructor and "step()" API.
RPC
Before using RPC and distributed autograd primitives, initialization
must take place. To initialize the RPC framework we need to use
"init_rpc()" which would initialize the RPC framework, RRef framework
and distributed autograd.
torch.distributed.rpc.init_rpc(name, backend=None, rank=- 1, world_size=None, rpc_backend_options=None)
Initializes RPC primitives such as the local RPC agent and
   distributed autograd, which immediately makes the current process
   ready to send and receive RPCs.
Parameters:
ready to send and receive RPCs.
Parameters:
      * name (str) -- a globally unique name of this node.
        (e.g., "Trainer3", "ParameterServer2", "Master", "Worker1")
        Name can only contain number, alphabet, underscore, colon,
        and/or dash, and must be shorter than 128 characters.
  * **backend** (*BackendType**, **optional*) -- The type of RPC
    backend implementation. Supported values is
    "BackendType.TENSORPIPE" (the default). See Backends for more
    information.

  * **rank** (*int*) -- a globally unique id/rank of this node.

  * **world_size** (*int*) -- The number of workers in the group.

  * **rpc_backend_options** (*RpcBackendOptions**, **optional*) --
    The options passed to the RpcAgent constructor. It must be an
    agent-specific subclass of "RpcBackendOptions" and contains
    agent-specific initialization configurations. By default, for
    all agents, it sets the default timeout to 60 seconds and
performs the rendezvous with an underlying process group
        initialized using "init_method = "env://"", meaning that
        environment variables "MASTER_ADDR" and "MASTER_PORT" need to
        be set properly. See Backends for more information and find
        which options are available.
The following APIs allow users to remotely execute functions as well
as create references (RRefs) to remote data objects. In these APIs,
when passing a "Tensor" as an argument or a return value, the
destination worker will try to create a "Tensor" with the same meta
(i.e., shape, stride, etc.). We intentionally disallow transmitting
CUDA tensors because it might crash if the device lists on source and
destination workers do not match. In such cases, applications can
always explicitly move the input tensors to CPU on the caller and move
it to the desired devices on the callee if necessary.
Warning:
TorchScript support in RPC is a prototype feature and subject to
change. Since v1.5.0, "torch.distributed.rpc" supports calling
  TorchScript functions as RPC target functions, and this will help
  improve parallelism on the callee side as executing TorchScript
  functions does not require GIL.
torch.distributed.rpc.rpc_sync(to, func, args=None, kwargs=None, timeout=- 1.0)
Make a blocking RPC call to run function "func" on worker "to". RPC
   messages are sent and received in parallel to execution of Python
   code. This method is thread-safe.
Parameters:
      * to (str or WorkerInfo or int) --
        name/rank/"WorkerInfo" of the destination worker.
  * **func** (*Callable*) -- a callable function, such as Python
    callables, builtin operators (e.g. "add()") and annotated
    TorchScript functions.

  * **args** (*tuple*) -- the argument tuple for the "func"
    invocation.

  * **kwargs** (*dict*) -- is a dictionary of keyword arguments
    for the "func" invocation.
for the "func" invocation.
  * **timeout** (*float**, **optional*) -- timeout in seconds to
    use for this RPC. If the RPC does not complete in this amount
    of time, an exception indicating it has timed out will be
    raised. A value of 0 indicates an infinite timeout, i.e. a
    timeout error will never be raised. If not provided, the
    default value set during initialization or with
    "_set_rpc_timeout" is used.

Returns:
      Returns the result of running "func" with "args" and "kwargs".
Example::
      Make sure that "MASTER_ADDR" and "MASTER_PORT" are set properly
      on both workers. Refer to "init_process_group()" API for more
      details. For example,
  export MASTER_ADDR=localhost export MASTER_PORT=5678

  Then run the following code in two different processes:

  >>> # On worker 0:
  >>> import torch
  >>> import torch.distributed.rpc as rpc
  >>> rpc.init_rpc("worker0", rank=0, world_size=2)
ret = rpc.rpc_sync("worker1", torch.add, args=(torch.ones(2), 3))
      >>> rpc.shutdown()



  >>> # On worker 1:
  >>> import torch.distributed.rpc as rpc
  >>> rpc.init_rpc("worker1", rank=1, world_size=2)
  >>> rpc.shutdown()

  Below is an example of running a TorchScript function using RPC.

  >>> # On both workers:
  >>> @torch.jit.script
  >>> def my_script_add(t1, t2):
  >>>    return torch.add(t1, t2)

  >>> # On worker 0:
  >>> import torch.distributed.rpc as rpc
  >>> rpc.init_rpc("worker0", rank=0, world_size=2)
  >>> ret = rpc.rpc_sync("worker1", my_script_add, args=(torch.ones(2), 3))
  >>> rpc.shutdown()

  >>> # On worker 1:
  >>> import torch.distributed.rpc as rpc
  >>> rpc.init_rpc("worker1", rank=1, world_size=2)
  >>> rpc.shutdown()

torch.distributed.rpc.rpc_async(to, func, args=None, kwargs=None, timeout=- 1.0)
Make a non-blocking RPC call to run function "func" on worker "to".
RPC messages are sent and received in parallel to execution of
   Python code. This method is thread-safe. This method will
   immediately return a "Future" that can be awaited on.
Parameters:
      * to (str or WorkerInfo or int) --
        name/rank/"WorkerInfo" of the destination worker.
  * **func** (*Callable*) -- a callable function, such as Python
    callables, builtin operators (e.g. "add()") and annotated
    TorchScript functions.

  * **args** (*tuple*) -- the argument tuple for the "func"
    invocation.

  * **kwargs** (*dict*) -- is a dictionary of keyword arguments
    for the "func" invocation.

  * **timeout** (*float**, **optional*) -- timeout in seconds to
    use for this RPC. If the RPC does not complete in this amount
    of time, an exception indicating it has timed out will be
    raised. A value of 0 indicates an infinite timeout, i.e. a
    timeout error will never be raised. If not provided, the