shrinath-suresh/blogs-docs-splitted huggingface dataset:
internal optimization of DDP for future communication hook
   developers.
Parameters:
      * state (PowerSGDState) -- State information to configure
        the compression rate and support error feedback, warm start,
        etc. To tune the compression configs, mainly need to tune
        "matrix_approximation_rank", "start_powerSGD_iter" and
        "min_compression_rate".
  * **bucket** (*dist.GradBucket*) -- Bucket that stores a 1D
    flattened gradient tensor that batches multiple per-variable
    tensors. Note that since DDP comm hook only supports single
    process single device mode, only exactly one tensor is stored
    in this bucket.

Returns:
      Future handler of the communication, which updates the gradients
      in place.
Return type:
      Future[Tensor]
Example::
      >>> state = PowerSGDState(process_group=process_group, matrix_approximation_rank=1,
                                start_powerSGD_iter=10, min_compression_rate=0.5)
ddp_model.register_comm_hook(state, powerSGD_hook)



torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook.batched_powerSGD_hook(state, bucket)
This DDP communication hook implements a simplified PowerSGD
   gradient compression algorithm described in the paper. This variant
   does not compress the gradients layer by layer, but instead
   compresses the flattened input tensor that batches all the
   gradients. Therefore, it is faster than "powerSGD_hook()", but
   usually results in a much lower accuracy, unless
   "matrix_approximation_rank" is 1.
Warning:
 Increasing "matrix_approximation_rank" here may not necessarily
 increase the accuracy, because batching per-parameter tensors
 without column/row alignment can destroy low-rank structure.
 Therefore, the user should always consider "powerSGD_hook()"
 first, and only consider this variant when a satisfactory
 accuracy can be achieved when "matrix_approximation_rank" is 1.
Once gradient tensors are aggregated across all workers, this hook
   applies compression as follows:


Views the input flattened 1D gradient tensor as a square-shaped
      tensor M with 0 paddings;


Creates two low-rank tensors P and Q for decomposing M, such
      that M = PQ^T, where Q is initialized from a standard normal
      distribution and orthogonalized;


Computes P, which is equal to MQ;


Allreduces P;


Orthogonalizes P;


Computes Q, which is approximately equal to M^TP;


Allreduces Q;


Computes M, which is approximately equal to PQ^T.


Truncates the input tensor to the original length.


Note that this communication hook enforces vanilla allreduce for
   the first "state.start_powerSGD_iter" iterations. This not only
   gives the user more control over the tradeoff between speedup and
   accuracy, but also helps abstract away some complexity of the
   internal optimization of DDP for future communication hook
   developers.
developers.
Parameters:
      * state (PowerSGDState) -- State information to configure
        the compression rate and support error feedback, warm start,
        etc. To tune the compression configs, mainly need to tune
        "matrix_approximation_rank" and "start_powerSGD_iter".
  * **bucket** (*dist.GradBucket*) -- Bucket that stores a 1D
    flattened gradient tensor that batches multiple per-variable
    tensors. Note that since DDP comm hook only supports single
    process single device mode, only exactly one tensor is stored
    in this bucket.

Returns:
      Future handler of the communication, which updates the gradients
      in place.
Return type:
      Future[Tensor]
Example::
      >>> state = PowerSGDState(process_group=process_group, matrix_approximation_rank=1)
      >>> ddp_model.register_comm_hook(state, batched_powerSGD_hook)
Debugging Communication Hooks
=============================
As the name implies, debugging communication hooks are only used
for debugging and performance optimization purpose.
Warning:
Debugging communication hooks do not necessarily output the correct
  results.
torch.distributed.algorithms.ddp_comm_hooks.debugging_hooks.noop_hook(_, bucket)
This DDP communication hook returns a future that wraps the input,
   so it is a noop that does not incur any communication overheads.
This hook should only be used for headroom analysis of
   allreduce optimization, instead of the normal gradient
   synchronization. For example, if only less than 10% speedup of
   training time can be observed after this hook is registered, it
   usually implies that allreduce is not a performance bottleneck for
   this case. Such instrumentation can be particularly useful if GPU
   traces cannot be easily retrieved or the trace analysis is
   complicated some factors such as the overlap between allreduce and
computation or the desynchronization across ranks.
Example::
      >>> ddp_model.register_comm_hook(None, noop_hook)
Return type:
      Future[Tensor]
Checkpointing of Communication Hooks
A stateful communication hook can be saved as a part of model
checkpointing to enable trainer restarts. To make a hook serializable,
"setstate" and "getstate" should be defined.
Warning:
"getstate" should exclude non-serializable attributes from a
  returned dictionary.
Warning:
"setstate" should properly initialize non-serializable
  attributes, excluded from a provided "state".
"PowerSGDState" has "setstate" and "getstate" implemented and
can be used as a reference.
class torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook.PowerSGDState
getstate()
  Returns a "Dict[str, Any]" which will be pickled and saved.
  "process_group" is not serializable and excluded from a returned
  state.

setstate(state)
state.
setstate(state)
  Takes a provided "state" and retrieves "PowerSGDState".
  "process_group" is set to default.

Here is a simple, end-to-end example of saving and reloading PowerSGD
state and hook.
import os
   import sys
   import tempfile
   import torch
   import torch.distributed as dist
   import torch.nn as nn
   import torch.optim as optim
from torch.distributed.algorithms.ddp_comm_hooks import powerSGD_hook as powerSGD
class SimpleModel(nn.Module):
       def init(self):
           super(SimpleModel, self).init()
           self.fc1 = nn.Linear(24,24)
           self.relu = nn.ReLU()
           self.fc2 = nn.Linear(24,12)
   def forward(self, x):
       return self.fc2(self.relu(self.fc1(x)))

def setup(rank, world_size):
       os.environ['MASTER_ADDR'] = 'localhost'
       os.environ['MASTER_PORT'] = '12355'
   # initialize the process group
   dist.init_process_group("nccl", rank=rank, world_size=world_size)

def cleanup():
def cleanup():
       dist.destroy_process_group()
def run_demo(demo_fn, world_size):
       mp.spawn(
           demo_fn,
           args=(world_size,),
           nprocs=world_size,
           join=True)
def demo_serialization(rank, world_size):
       setup(rank, world_size)
   CHECKPOINT = tempfile.gettempdir() + "/checkpoint.pt"

   model = SimpleModel().to(rank)
   ddp_model = DistributedDataParallel(model, device_ids=[rank])

   powersgd_hook = powerSGD.powerSGD_hook
   powersgd_state = powerSGD.PowerSGDState(process_group=None)

   optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)
   ddp_model.register_comm_hook(powersgd_state, powersgd_hook)

   state = {
       'state_dict': ddp_model.state_dict(),
       'comm_hook': hook,
       'comm_hook_state': hook_state}

   if rank == 0:
       torch.save(state, CHECKPOINT)

   dist.barrier()
   map_location = {'cuda:%d' % 0: 'cuda:%d' % rank}
checkpoint = torch.load(CHECKPOINT, map_location=map_location)
   ddp_model.load_state_dict(checkpoint['state_dict'])
   powersgd_hook = checkpoint['comm_hook']
   powersgd_state = checkpoint['comm_hook_state']

   ddp_model.register_comm_hook(powersgd_state, powersgd_hook)

   if rank == 0:
       os.remove(CHECKPOINT)

   cleanup()

if name == "main":
       n_gpus = torch.cuda.device_count()
       assert n_gpus >= 2, f"Requires at least 2 GPUs to run, but got {n_gpus}"
       world_size = n_gpus
       run_demo(demo_serialization, world_size)
Acknowledgements
Many thanks to PowerSGD paper author Thijs Vogels for the code
review on PowerSGD communication hook, as well as the comparison
experiments, which show that the performance of PowerSGD communication
hook is on par with the implementation in the original paper.
Pipeline Parallelism
Pipeline parallelism was original introduced in the Gpipe  paper and
is an efficient technique to train large models on multiple GPUs.
Warning:
Pipeline Parallelism is experimental and subject to change.
Model Parallelism using multiple GPUs
Typically for large models which don't fit on a single GPU, model
parallelism is employed where certain parts of the model are placed on
different GPUs. Although, if this is done naively for sequential
models, the training process suffers from GPU under utilization since
only one GPU is active at one time as shown in the figure below:
[image]The figure represents a model with 4 layers placed on 4
   different GPUs (vertical axis). The horizontal axis represents
   training this model through time demonstrating that only 1 GPU is
   utilized at a time (image source).
Pipelined Execution
To alleviate this problem, pipeline parallelism splits the input
minibatch into multiple microbatches and pipelines the execution of
these microbatches across multiple GPUs. This is outlined in the
figure below:
[image]The figure represents a model with 4 layers placed on 4
   different GPUs (vertical axis). The horizontal axis represents
   training this model through time demonstrating that the GPUs are
   utilized much more efficiently. However, there still exists a
   bubble (as demonstrated in the figure) where certain GPUs are not
   utilized. (image source).
Pipe APIs in PyTorch
class torch.distributed.pipeline.sync.Pipe(module, chunks=1, checkpoint='except_last', deferred_batch_norm=False)
Wraps an arbitrary "nn.Sequential" module to train on using
   synchronous pipeline parallelism. If the module requires lots of
   memory and doesn't fit on a single GPU, pipeline parallelism is a
   useful technique to employ for training.
The implementation is based on the torchgpipe paper.
Pipe combines pipeline parallelism with checkpointing to reduce
   peak memory required to train while minimizing device under-
   utilization.
You should place all the modules on the appropriate devices and
   wrap them into an "nn.Sequential" module defining the desired order
   of execution. If a module does not contain any parameters/buffers,
   it is assumed this module should be executed on CPU and appropriate
   input tensors to the module are moved to CPU before execution. This
   behavior can be overridden by the "WithDevice" wrapper which can be
   used to explicitly specify which device a module should run on.
Parameters:
      * module ("nn.Sequential") -- sequential module to be
        parallelized using pipelining. Each module in the sequence has
        to have all of its parameters on a single device. Each module
        in the sequence has to either be an nn.Module or
        "nn.Sequential" (to combine multiple sequential modules on a
        single device)
single device)
  * **chunks** (*int*) -- number of micro-batches (default: "1")

  * **checkpoint** (*str*) -- when to enable checkpointing, one of
    "'always'", "'except_last'", or "'never'" (default:
    "'except_last'"). "'never'" disables checkpointing completely,
    "'except_last'" enables checkpointing for all micro-batches
    except the last one and "'always'" enables checkpointing for
    all micro-batches.

  * **deferred_batch_norm** (*bool*) -- whether to use deferred
    "BatchNorm" moving statistics (default: "False"). If set to
    "True", we track statistics across multiple micro-batches to
    update the running statistics per mini-batch.

Raises:
      * TypeError -- the module is not a "nn.Sequential".
  * **ValueError** -- invalid arguments

Example::
      Pipeline of two FC layers across GPUs 0 and 1.
  >>> # Need to initialize RPC framework first.
  >>> os.environ['MASTER_ADDR'] = 'localhost'
os.environ['MASTER_ADDR'] = 'localhost'
      >>> os.environ['MASTER_PORT'] = '29500'
      >>> torch.distributed.rpc.init_rpc('worker', rank=0, world_size=1)
      >>>
      >>> # Build pipe.
      >>> fc1 = nn.Linear(16, 8).cuda(0)
      >>> fc2 = nn.Linear(8, 4).cuda(1)
      >>> model = nn.Sequential(fc1, fc2)
      >>> model = Pipe(model, chunks=8)
      >>> input = torch.rand(16, 16).cuda(0)
      >>> output_rref = model(input)



Note:
 You can wrap a "Pipe" model with
 "torch.nn.parallel.DistributedDataParallel" only when the
 checkpoint parameter of "Pipe" is "'never'".

Note:
 "Pipe" only supports intra-node pipelining currently, but will be
 expanded to support inter-node pipelining in the future. The
 forward function returns an "RRef" to allow for inter-node
 pipelining in the future, where the output might be on a remote
 host. For intra-node pipelinining you can use "local_value()" to
 retrieve the output locally.

Warning:
retrieve the output locally.
Warning:
 "Pipe" is experimental and subject to change.

forward(*inputs)
  Processes a single input mini-batch through the pipe and returns
  an "RRef" pointing to the output. "Pipe" is a fairly transparent
  module wrapper. It doesn't modify the input and output signature
  of the underlying module. But there's type restriction. Input
  and output have to contain at least one tensor. This restriction
  is applied at partition boundaries too.

  The sequence of inputs are fed into the first stage of the
  pipeline as "*inputs". As a result the positional args for this
  function should match the positional args for the first stage of
  the pipeline. The same condition applies for output of one stage
  of the pipeline which is the input for the next stage.

  The input tensor is split into multiple micro-batches based on
  the "chunks" parameter used to initialize "Pipe". The batch size
is assumed to be the first dimension of the tensor and if the
      batch size is less than "chunks", the number of micro-batches is
      equal to the batch size.
  Only tensors are split into multiple micro-batches, non-Tensor
  inputs are just replicated as-is in each micro-batch. For non-
  Tensor outputs in the last stage of the pipeline, they are
  aggregated as a "List" and returned the user. For example, if
  you have 2 micro-batches returning the integer 5, the user would
  receive the consolidated output of *[5, 5]*

  All the input tensors need to be on the same device as the first
  partition of the pipeline.

  If a tensor is wrapped with the "NoChunk" wrapper, the tensor is
  not split across micro-batches and is replicated as-is similar
  to non-tensors.

  Parameters:
     **inputs** -- input mini-batch

  Returns:
     "RRef" to the output of the mini-batch

  Raises:
Raises:
         TypeError -- input doesn't contain at least one tensor
  Return type:
     *RRef*

Skip connections
Certain models like ResNeXt are not completely sequential and have
skip connections between layers. Naively implementing as part of
pipeline parallelism would imply that we need to copy outputs for
certain layers through multiple GPUs till we eventually reach the GPU
where the layer for the skip connection resides. To avoid this copy
overhead, we provide APIs below to stash and pop Tensors in different
layers of the model.
torch.distributed.pipeline.sync.skip.skippable.skippable(stash=(), pop=())
The decorator to define a "nn.Module" with skip connections.
   Decorated modules are called "skippable". This functionality works
   perfectly fine even when the module is not wrapped by "Pipe".
Each skip tensor is managed by its name. Before manipulating skip
   tensors, a skippable module must statically declare the names for
skip tensors by stash and/or pop parameters. Skip tensors with
   pre-declared name can be stashed by "yield stash(name, tensor)" or
   popped by "tensor = yield pop(name)".
Here is an example with three layers. A skip tensor named "1to3" is
   stashed and popped at the first and last layer, respectively:
  @skippable(stash=['1to3'])
  class Layer1(nn.Module):
      def forward(self, input):
          yield stash('1to3', input)
          return f1(input)

  class Layer2(nn.Module):
      def forward(self, input):
          return f2(input)

  @skippable(pop=['1to3'])
  class Layer3(nn.Module):
      def forward(self, input):
          skip_1to3 = yield pop('1to3')
          return f3(input) + skip_1to3

  model = nn.Sequential(Layer1(), Layer2(), Layer3())

One skippable module can stash or pop multiple skip tensors:
  @skippable(stash=['alice', 'bob'], pop=['carol'])
  class StashStashPop(nn.Module):
class StashStashPop(nn.Module):
          def forward(self, input):
              yield stash('alice', f_alice(input))
              yield stash('bob', f_bob(input))
              carol = yield pop('carol')
              return input + carol
Every skip tensor must be associated with exactly one pair of
   stash and pop. "Pipe" checks this restriction automatically
   when wrapping a module. You can also check the restriction by
   "verify_skippables()" without "Pipe".
Return type:
      Callable[[Type[Module]], Type[Skippable]]
class torch.distributed.pipeline.sync.skip.skippable.stash(name, tensor)
The command to stash a skip tensor.
  def forward(self, input):
      yield stash('name', input)
      return f(input)

Parameters:
      * name (str) -- name of skip tensor
  * **input** (*torch.Tensor** or **None*) -- tensor to pass to
    the skip connection

class torch.distributed.pipeline.sync.skip.skippable.pop(name)
The command to pop a skip tensor.
  def forward(self, input):
      skip = yield pop('name')
      return f(input) + skip

Parameters:
      name (str) -- name of skip tensor
Returns:
      the skip tensor previously stashed by another layer under the
      same name
Return type:
      None
torch.distributed.pipeline.sync.skip.skippable.verify_skippables(module)
Verifies if the underlying skippable modules satisfy integrity.
Every skip tensor must have only one pair of stash and pop. If
   there are one or more unmatched pairs, it will raise "TypeError"
   with the detailed messages.
Here are a few failure cases. "verify_skippables()" will report
   failure for these cases:
  # Layer1 stashes "1to3".
  # Layer3 pops "1to3".

  nn.Sequential(Layer1(), Layer2())
  #               âââââ ?

  nn.Sequential(Layer2(), Layer3())
  #                   ? âââââ

  nn.Sequential(Layer1(), Layer2(), Layer3(), Layer3())
âââââââââââââââââââââ       ^^^^^^
  nn.Sequential(Layer1(), Layer1(), Layer2(), Layer3())
  #             ^^^^^^      âââââââââââââââââââââ

To use the same name for multiple skip tensors, they must be
   isolated by different namespaces. See "isolate()".
Raises:
      TypeError -- one or more pairs of stash and pop are not
      matched.
Tutorials
The following tutorials give a good overview of how to use the "Pipe"
API to train your models with the rest of the components that PyTorch
provides:


Training Transformer models using Pipeline Parallelism


Training Transformer models using Distributed Data Parallel and
  Pipeline Parallelism


Acknowledgements
The implementation for pipeline parallelism is based on fairscale's
pipe implementation and torchgpipe. We would like to thank both teams
torch.backends
torch.backends controls the behavior of various backends that
PyTorch supports.
These backends include:


"torch.backends.cuda"


"torch.backends.cudnn"


"torch.backends.mps"


"torch.backends.mkl"


"torch.backends.mkldnn"


"torch.backends.openmp"


"torch.backends.opt_einsum"


"torch.backends.xeon"


torch.backends.cuda
torch.backends.cuda.is_built()
Returns whether PyTorch is built with CUDA support.  Note that this
   doesn't necessarily mean CUDA is available; just that if this
   PyTorch binary were run a machine with working CUDA drivers and
   devices, we would be able to use it.
torch.backends.cuda.matmul.allow_tf32
A "bool" that controls whether TensorFloat-32 tensor cores may be
   used in matrix multiplications on Ampere or newer GPUs. See
   TensorFloat-32(TF32) on Ampere devices.
torch.backends.cuda.matmul.allow_fp16_reduced_precision_reduction
A "bool" that controls whether reduced precision reductions (e.g.,
with fp16 accumulation type) are allowed with fp16 GEMMs.
torch.backends.cuda.matmul.allow_bf16_reduced_precision_reduction
A "bool" that controls whether reduced precision reductions are
   allowed with bf16 GEMMs.
torch.backends.cuda.cufft_plan_cache
"cufft_plan_cache" caches the cuFFT plans
size
  A readonly "int" that shows the number of plans currently in the
  cuFFT plan cache.

torch.backends.cuda.max_size
  A "int" that controls cache capacity of cuFFT plan.

torch.backends.cuda.clear()
  Clears the cuFFT plan cache.

torch.backends.cuda.preferred_linalg_library(backend=None)
Warning:
 This flag is experimental and subject to change.

When PyTorch runs a CUDA linear algebra operation it often uses the
   cuSOLVER or MAGMA libraries, and if both are available it decides
   which to use with a heuristic. This flag (a "str") allows
   overriding those heuristics.

If "cusolver" is set then cuSOLVER will be used wherever
     possible.
possible.


If "magma" is set then MAGMA will be used wherever possible.


If "default" (the default) is set then heuristics will be used
     to pick between cuSOLVER and MAGMA if both are available.


When no input is given, this function returns the currently
     preferred library.


Note: When a library is preferred other libraries may still be used
   if the preferred library doesn't implement the operation(s) called.
   This flag may achieve better performance if PyTorch's heuristic
   library selection is incorrect for your application's inputs.
Currently supported linalg operators:


"torch.linalg.inv()"


"torch.linalg.inv_ex()"


"torch.linalg.cholesky()"


"torch.linalg.cholesky_ex()"


"torch.cholesky_solve()"


"torch.cholesky_inverse()"


"torch.linalg.lu_factor()"


"torch.linalg.lu()"


"torch.linalg.lu_solve()"


"torch.linalg.qr()"


"torch.linalg.eigh()"


"torch.linalg.eighvals()"


"torch.linalg.svd()"
"torch.linalg.svd()"


"torch.linalg.svdvals()"


Return type:
      _LinalgBackend
class torch.backends.cuda.SDPBackend(value)
Enum class for the scaled dot product attention backends.
Warning:
 This flag is experimental and subject to change.'

This class needs to stay inline with the enum defined in:
   pytorch/aten/src/ATen/native/transformers/sdp_utils_cpp.h
torch.backends.cuda.flash_sdp_enabled()
Warning:
 This flag is experimental and subject to change.

Returns whether flash sdp is enabled or not.
torch.backends.cuda.enable_mem_efficient_sdp(enabled)
Warning:
 This flag is experimental and subject to change.

Enables or disables memory efficient sdp.
torch.backends.cuda.mem_efficient_sdp_enabled()
Warning:
 This flag is experimental and subject to change.

Returns whether memory efficient sdp is enabled or not.
torch.backends.cuda.enable_flash_sdp(enabled)
Warning:
 This flag is experimental and subject to change.
Enables or disables flash sdp.
torch.backends.cuda.math_sdp_enabled()
Warning:
 This flag is experimental and subject to change.

Returns whether math sdp is enabled or not.
torch.backends.cuda.enable_math_sdp(enabled)
Warning:
 This flag is experimental and subject to change.

Enables or disables math sdp.
torch.backends.cuda.sdp_kernel(enable_flash=True, enable_math=True, enable_mem_efficient=True)
Warning:
 This flag is experimental and subject to change.

This context manager can be used to temporarily enable or disable
   flash/memory efficient sdp and math sdp. Upon exiting the context
   manager, the previous state of the flags will be restored.
torch.backends.cudnn
torch.backends.cudnn.version()
Returns the version of cuDNN
torch.backends.cudnn.is_available()
Returns a bool indicating if CUDNN is currently available.
torch.backends.cudnn.enabled
A "bool" that controls whether cuDNN is enabled.