shrinath-suresh/blogs-docs-splitted huggingface dataset:
can lead to wrong results or DDP backward hang.
Implementation
Below are pointers to the DDP implementation components. The stacked
graph shows the structure of the code.
ProcessGroup


ProcessGroup.hpp: contains the abstract API of all process group
  implementations. The "c10d" library provides 3 implementations out
  of the box, namely, ProcessGroupGloo, ProcessGroupNCCL, and
  ProcessGroupMPI. "DistributedDataParallel" uses
  "ProcessGroup::broadcast()" to send model states from the process
  with rank 0 to others during initialization and
  "ProcessGroup::allreduce()" to sum gradients.


Store.hpp: assists the rendezvous service for process group
  instances to find each other.


DistributedDataParallel

distributed.py: is the Python entry point for DDP. It implements the
  initialization steps and the "forward" function for the
  "nn.parallel.DistributedDataParallel" module which call into C++
libraries. Its "_sync_param" function performs intra-process
  parameter synchronization when one DDP process works on multiple
  devices, and it also broadcasts model buffers from the process with
  rank 0 to all other processes. The inter-process parameter
  synchronization happens in "Reducer.cpp".


comm.h: implements the coalesced broadcast helper function which is
  invoked to broadcast model states during initialization and
  synchronize model buffers before the forward pass.


reducer.h: provides the core implementation for gradient
  synchronization in the backward pass. It has three entry point
  functions:


"Reducer": The constructor is called in "distributed.py" which
    registers "Reducer::autograd_hook()" to gradient accumulators.


"autograd_hook()" function will be invoked by the autograd engine
    when a gradient becomes ready.


"prepare_for_backward()" is called at the end of DDP forward pass
    in "distributed.py". It traverses the autograd graph to find
unused parameters when "find_unused_parameters" is set to "True"
    in DDP constructor.
[image: ddp_code.png][image]
TorchDynamo DDPOptimizer
DDP's performance advantage comes from overlapping allreduce
collectives with computations during backwards. AotAutograd prevents
this overlap when used with TorchDynamo for compiling a whole forward
and whole backward graph, because allreduce ops are launched by
autograd hooks after the whole optimized backwards computation
finishes.
TorchDynamo's DDPOptimizer helps by breaking the forward graph at the
logical boundaries of DDP's allreduce buckets during backwards.  Note:
the goal is to break the graph during backwards, and the simplest
implementation is to break the forward graphs and then call
AotAutograd and compilation on each section.  This allows DDP's
allreduce hooks to fire in-between sections of backwards, and schedule
communications to overlap with compute.
communications to overlap with compute.
See this blog post for a more in-depth explanation and experimental
results, or read the docs and code at
torch/_dynamo/optimizations/distributed.py
To Debug DDPOptimizer, set torch._dynamo.config.log_level to DEBUG
(for full graph dumps) or INFO (for basic info about bucket
boundaries).  To disable DDPOptimizer, set
torch._dynamo.config.optimize_ddp=False. DDP and TorchDynamo should
still work correctly without DDPOptimizer, but with performance
degradation.
Features for large-scale deployments
* Fleet-wide operator profiling


API usage logging


Attaching metadata to saved TorchScript models


Build environment considerations


Common extension points


This note talks about several extension points and tricks that might
be useful when running PyTorch within a larger system or operating
multiple systems using PyTorch in a larger organization.
It doesn't cover topics of deploying models to production. Check
"torch.jit" or one of the corresponding tutorials.
The note assumes that you either build PyTorch from source in your
organization or have an ability to statically link additional code to
be loaded when PyTorch is used. Therefore, many of the hooks are
exposed as C++ APIs that can be triggered once in a centralized place,
e.g. in static initialization code.
Fleet-wide operator profiling
PyTorch comes with "torch.autograd.profiler" capable of measuring time
taken by individual operators on demand. One can use the same
mechanism to do "always ON" measurements for any process running
PyTorch. It might be useful for gathering information about PyTorch
workloads running in a given process or across the entire set of
machines.
New callbacks for any operator invocation can be added with
"torch::addGlobalCallback". Hooks will be called with
"torch::RecordFunction" struct that describes invocation context (e.g.
name). If enabled, "RecordFunction::inputs()" contains arguments of
the function represented as "torch::IValue" variant type. Note, that
inputs logging is relatively expensive and thus has to be enabled
explicitly.
The operator callbacks also have access to
"c10::ThreadLocalDebugInfo::get()" interface that returns a pointer to
the struct holding the debug information. This debug information can
be set earlier by using "at::DebugInfoGuard" object. Debug information
is propagated through the forward (including async "fork" tasks) and
backward passes and can be useful for passing some extra information
about execution environment (e.g. model id) from the higher layers of
the application down to the operator callbacks.
Invoking callbacks adds some overhead, so usually it's useful to just
randomly sample operator invocations. This can be enabled on per-
callback basis with an optional sampling rate passed into
"torch::addGlobalCallback".
Note, that "addGlobalCallback" is not thread-safe and can be called
only when no PyTorch operator is running. Usually, it's a good idea to
call them once during initialization.
Here's an example:
// Called somewhere in the program beginning
   void init() {
       // Sample one in a hundred operator runs randomly
       addGlobalCallback(
         RecordFunctionCallback(
           &onFunctionEnter,
           &onFunctionExit)
         .needsInputs(true)
         .samplingProb(0.01)
       );
       // Note, to enable observers in the model calling thread,
// call enableRecordFunction() in the thread before running a model
   }
void onFunctionEnter(const RecordFunction& fn) {
       std::cerr << "Before function " << fn.name()
                 << " with " << fn.inputs().size() << " inputs" << std::endl;
   }
void onFunctionExit(const RecordFunction& fn) {
       std::cerr << "After function " << fn.name();
   }
API usage logging
When running in a broader ecosystem, for example in managed job
scheduler, it's often useful to track which binaries invoke particular
PyTorch APIs. There exists simple instrumentation injected at several
important API points that triggers a given callback. Because usually
PyTorch is invoked in one-off python scripts, the callback fires only
once for a given process for each of the APIs.
"c10::SetAPIUsageHandler" can be used to register API usage
instrumentation handler. Passed argument is going to be an "api key"
identifying used point, for example "python.import" for PyTorch
extension import or "torch.script.compile" if TorchScript compilation
was triggered.
SetAPIUsageLogger( {
       std::cerr << "API was used: " << event_name << std::endl;
   });
Note for developers: new API trigger points can be added in code with
"C10_LOG_API_USAGE_ONCE("my_api")" in C++ or
"torch._C._log_api_usage_once("my.api")" in Python.
Attaching metadata to saved TorchScript models
TorchScript modules can be saved as an archive file that bundles
serialized parameters and module code as TorchScript (see
"torch.jit.save()"). It's often convenient to bundle additional
information together with the model, for example, description of model
producer or auxiliary artifacts.
It can be achieved by passing the "_extra_files" argument to
"torch.jit.save()" and "torch::jit::load" to store and retrieve
arbitrary binary blobs during saving process. Since TorchScript files
are regular ZIP archives, extra information gets stored as regular
files inside archive's "extra/" directory.
There's also a global hook allowing to attach extra files to any
TorchScript archive produced in the current process. It might be
useful to tag models with producer metadata, akin to JPEG metadata
produced by digital cameras. Example usage might look like:
SetExportModuleExtraFilesHook( {
       ExtraFilesMap files;
       files["producer_info.json"] = "{\"user\": \"" + getenv("USER") + "\"}";
       return files;
   });
Build environment considerations
TorchScript's compilation needs to have access to the original python
files as it uses python's "inspect.getsource" call. In certain
production environments it might require explicitly deploying ".py"
files along with precompiled ".pyc".
Common extension points
PyTorch APIs are generally loosely coupled and it's easy to replace a
Numerical accuracy
In modern computers, floating point numbers are represented using IEEE
754 standard. For more details on floating point arithmetics and IEEE
754 standard, please see Floating point arithmetic In particular, note
that floating point provides limited accuracy (about 7 decimal digits
for single precision floating point numbers, about 16 decimal digits
for double precision floating point numbers) and that floating point
addition and multiplication are not associative, so the order of the
operations affects the results. Because of this, PyTorch is not
guaranteed to produce bitwise identical results for floating point
computations that are mathematically identical. Similarly, bitwise
identical results are not guaranteed across PyTorch releases,
individual commits, or different platforms. In particular, CPU and GPU
results can be different even for bitwise-identical inputs and even
after controlling for the sources of randomness.
Batched computations or slice computations
Batched computations or slice computations
Many operations in PyTorch support batched computation, where the same
operation is performed for the elements of the batches of inputs. An
example of this is "torch.mm()" and "torch.bmm()". It is possible to
implement batched computation as a loop over batch elements, and apply
the necessary math operations to the individual batch elements, for
efficiency reasons we are not doing that, and typically perform
computation for the whole batch. The mathematical libraries that we
are calling, and PyTorch internal implementations of operations can
produces slightly different results in this case, compared to non-
batched computations. In particular, let "A" and "B" be 3D tensors
with the dimensions suitable for batched matrix multiplication. Then
"(A@B)[0]" (the first element of the batched result) is not guaranteed
to be bitwise identical to "A[0]@B[0]" (the matrix product of the
first elements of the input batches) even though mathematically it's
an identical computation.
Similarly, an operation applied to a tensor slice is not guaranteed to
produce results that are identical to the slice of the result of the
same operation applied to the full tensor. E.g. let "A" be a
2-dimensional tensor. "A.sum(-1)[0]" is not guaranteed to be bitwise
equal to "A[:,0].sum()".
Extremal values
When inputs contain large values such that intermediate results may
overflow the range of the used datatype, the end result may overflow
too, even though it is representable in the original datatype. E.g.:
import torch
   a=torch.tensor([1e20, 1e20]) # fp32 type by default
   a.norm() # produces tensor(inf)
   a.double().norm() # produces tensor(1.4142e+20, dtype=torch.float64), representable in fp32
Linear algebra ("torch.linalg")
Non-finite values
The external libraries (backends) that "torch.linalg" uses provide no
guarantees on their behaviour when the inputs have non-finite values
like "inf" or "NaN". As such, neither does PyTorch. The operations may
return a tensor with non-finite values, or raise an exception, or even
segfault.
Consider using "torch.isfinite()" before calling these functions to
detect this situation.
Extremal values in linalg
Functions within "torch.linalg" have more Extremal Values than other
PyTorch functions.
Solvers and Inverses assume that the input matrix "A" is invertible.
If it is close to being non-invertible (for example, if it has a very
small singular value), then these algorithms may silently return
incorrect results. These matrices are said to be ill-conditioned. If
provided with ill-conditioned inputs, the result of these functions
they may vary when using the same inputs on different devices or when
using different backends via the keyword "driver".
Spectral operations like "svd", "eig", and "eigh" may also return
incorrect results (and their gradients may be infinite) when their
inputs have singular values that are close to each other. This is
because the algorithms used to compute these decompositions struggle
to converge for these inputs.
Running the computation in "float64" (as NumPy does by default) often
helps, but it does not solve these issues in all cases. Analyzing the
spectrum of the inputs via "torch.linalg.svdvals()" or their condition
number via "torch.linalg.cond()" may help to detect these issues.
TensorFloat-32(TF32) on Nvidia Ampere devices
On Ampere Nvidia GPUs, PyTorch can use TensorFloat32 (TF32) to speed
up mathematically intensive operations, in particular matrix
multiplications and convolutions. When an operation is performed using
TF32 tensor cores, only the first 10 bits of the input mantissa are
read. This may reduce accuracy and produce surprising results (e.g.,
multiplying a matrix by the identity matrix may produce results that
are different from the input). By default, TF32 tensor cores are
disabled for matrix multiplications and enabled for convolutions,
although most neural network workloads have the same convergence
behavior when using TF32 as they have with fp32. We recommend enabling
TF32 tensor cores for matrix multiplications with
"torch.backends.cuda.matmul.allow_tf32 = True" if your network does
not need full float32 precision. If your network needs full float32
precision for both matrix multiplications and convolutions, then TF32
tensor cores can also be disabled for convolutions with
"torch.backends.cudnn.allow_tf32 = False".
For more information see TensorFloat32.
Reduced Precision Reduction for FP16  and BF16 GEMMs
Half-precision GEMM operations are typically done with intermediate
accumulations (reduction) in single-precision for numerical accuracy
and improved resilience to overflow. For performance, certain GPU
architectures, especially more recent ones, allow a few truncations of
the intermediate accumulation results to the reduced precision (e.g.,
half-precision). This change is often benign from the perspective of
model convergence, though it may lead to unexpected results (e.g.,
"inf" values when the final result should be be representable in half-
precision). If reduced-precision reductions are problematic, they can
be turned off with
"torch.backends.cuda.matmul.allow_fp16_reduced_precision_reduction =
False"
A similar flag exists for BF16 GEMM operations and is turned off by
default. If BF16 reduced-precision reductions are problematic, they
can be turned off with
"torch.backends.cuda.matmul.allow_bf16_reduced_precision_reduction =
False"
For more information see allow_fp16_reduced_precision_reduction and
allow_bf16_reduced_precision_reduction
Reduced Precision FP16 and BF16 GEMMs and Convolutions on AMD Instinct MI200 devices
====================================================================================
On AMD Instinct MI200 GPUs, the FP16 and BF16 V_DOT2 and MFMA matrix
instructions flush input and output denormal values to zero. FP32 and
FP64 MFMA matrix instructions do not flush input and output denormal
values to zero. The affected instructions are only used by rocBLAS
(GEMM) and MIOpen (convolution) kernels; all other PyTorch operations
will not encounter this behavior. All other supported AMD GPUs will
not encounter this behavior.
rocBLAS and MIOpen provide alternate implementations for affected FP16
operations. Alternate implementations for BF16 operations are not
provided; BF16 numbers have a larger dynamic range than FP16 numbers
and are less likely to encounter denormal values. For the FP16
alternate implementations, FP16 input values are cast to an
intermediate BF16 value and then cast back to FP16 output after the
accumulate FP32 operations. In this way, the input and output types
are unchanged.
are unchanged.
When training using FP16 precision, some models may fail to converge
with FP16 denorms flushed to zero. Denormal values more frequently
occur in the backward pass of training during gradient calculation.
PyTorch by default will use the rocBLAS and MIOpen alternate
implementations during the backward pass. The default behavior can be
overridden using environment variables, ROCBLAS_INTERNAL_FP16_ALT_IMPL
and MIOPEN_DEBUG_CONVOLUTION_ATTRIB_FP16_ALT_IMPL. The behavior of
these environment variables is as follows:
+-----------------+-------------+-------------+
|                 | forward     | backward    |
|=================|=============|=============|
| Env unset       | original    | alternate   |
+-----------------+-------------+-------------+
| Env set to 1    | alternate   | alternate   |
+-----------------+-------------+-------------+
| Env set to 0    | original    | original    |
+-----------------+-------------+-------------+
+-----------------+-------------+-------------+
The following is the list of operations where rocBLAS may be used:


torch.addbmm


torch.addmm


torch.baddbmm


torch.bmm


torch.mm


torch.nn.GRUCell


torch.nn.LSTMCell


torch.nn.Linear


torch.sparse.addmm


the following torch._C._ConvBackend implementations:


slowNd


slowNd_transposed


slowNd_dilated


slowNd_dilated_transposed


The following is the list of operations where MIOpen may be used:


torch.nn.Conv[Transpose]Nd


the following torch._C._ConvBackend implementations:


ConvBackend::Miopen


ConvBackend::MiopenDepthwise


ConvBackend::MiopenTranspose
Broadcasting semantics
Many PyTorch operations support NumPy's broadcasting semantics. See
https://numpy.org/doc/stable/user/basics.broadcasting.html for
details.
In short, if a PyTorch operation supports broadcast, then its Tensor
arguments can be automatically expanded to be of equal sizes (without
making copies of the data).
General semantics
Two tensors are "broadcastable" if the following rules hold:


Each tensor has at least one dimension.


When iterating over the dimension sizes, starting at the trailing
  dimension, the dimension sizes must either be equal, one of them is
  1, or one of them does not exist.


For Example:



x=torch.empty(5,7,3)
y=torch.empty(5,7,3)
   # same shapes are always broadcastable (i.e. the above rules always hold)
x=torch.empty((0,))
y=torch.empty(2,2)
   # x and y are not broadcastable, because x does not have at least 1 dimension



# can line up trailing dimensions



x=torch.empty(5,3,4,1)
x=torch.empty(5,3,4,1)
y=torch.empty(  3,1,1)
   # x and y are broadcastable.
   # 1st trailing dimension: both have size 1
   # 2nd trailing dimension: y has size 1
   # 3rd trailing dimension: x size == y size
   # 4th trailing dimension: y dimension doesn't exist



# but:



x=torch.empty(5,2,4,1)
y=torch.empty(  3,1,1)
   # x and y are not broadcastable, because in the 3rd trailing dimension 2 != 3



If two tensors "x", "y" are "broadcastable", the resulting tensor size
is calculated as follows:


If the number of dimensions of "x" and "y" are not equal, prepend 1
  to the dimensions of the tensor with fewer dimensions to make them
  equal length.


Then, for each dimension size, the resulting dimension size is the
  max of the sizes of "x" and "y" along that dimension.


For Example:
# can line up trailing dimensions to make reading easier



x=torch.empty(5,1,4,1)
y=torch.empty(  3,1,1)
(x+y).size()
   torch.Size([5, 3, 4, 1])



# but not necessary:
but not necessary:



x=torch.empty(1)
y=torch.empty(3,1,7)
(x+y).size()
   torch.Size([3, 1, 7])
x=torch.empty(5,2,4,1)
y=torch.empty(3,1,1)
(x+y).size()
   RuntimeError: The size of tensor a (2) must match the size of tensor b (3) at non-singleton dimension 1



In-place semantics
One complication is that in-place operations do not allow the in-place
tensor to change shape as a result of the broadcast.
For Example:



x=torch.empty(5,3,4,1)
y=torch.empty(3,1,1)
(x.add_(y)).size()
   torch.Size([5, 3, 4, 1])



# but:



x=torch.empty(1,3,1)
y=torch.empty(3,1,7)
(x.add_(y)).size()
   RuntimeError: The expanded size of the tensor (1) must match the existing size (7) at non-singleton dimension 2.



Backwards compatibility
Prior versions of PyTorch allowed certain pointwise functions to
execute on tensors with different shapes, as long as the number of
elements in each tensor was equal.  The pointwise operation would then
be carried out by viewing each tensor as 1-dimensional.  PyTorch now
supports broadcasting and the "1-dimensional" pointwise behavior is
considered deprecated and will generate a Python warning in cases
where tensors are not broadcastable, but have the same number of
elements.
Note that the introduction of broadcasting can cause backwards
incompatible changes in the case where two tensors do not have the
same shape, but are broadcastable and have the same number of
elements. For Example:



torch.add(torch.ones(4,1), torch.randn(4))



would previously produce a Tensor with size: torch.Size([4,1]), but
now produces a Tensor with size: torch.Size([4,4]). In order to help
identify cases in your code where backwards incompatibilities
introduced by broadcasting may exist, you may set
torch.utils.backcompat.broadcast_warning.enabled to True, which
will generate a python warning in such cases.
For Example:
HIP (ROCm) semantics
ROCmâ¢ is AMDâs open source software platform for GPU-accelerated high
performance computing and machine learning. HIP is ROCm's C++ dialect
designed to ease conversion of CUDA applications to portable C++ code.
HIP is used when converting existing CUDA applications like PyTorch to
portable C++ and for new projects that require portability between AMD
and NVIDIA.
HIP Interfaces Reuse the CUDA Interfaces
PyTorch for HIP intentionally reuses the existing "torch.cuda"
interfaces. This helps to accelerate the porting of existing PyTorch
code and models because very few code changes are necessary, if any.
The example from CUDA semantics will work exactly the same for HIP:
cuda = torch.device('cuda')     # Default HIP device
   cuda0 = torch.device('cuda:0')  # 'rocm' or 'hip' are not valid, use 'cuda'
   cuda2 = torch.device('cuda:2')  # GPU 2 (these are 0-indexed)
x = torch.tensor([1., 2.], device=cuda0)
x = torch.tensor([1., 2.], device=cuda0)
   # x.device is device(type='cuda', index=0)
   y = torch.tensor([1., 2.]).cuda()
   # y.device is device(type='cuda', index=0)
with torch.cuda.device(1):
       # allocates a tensor on GPU 1
       a = torch.tensor([1., 2.], device=cuda)
   # transfers a tensor from CPU to GPU 1
   b = torch.tensor([1., 2.]).cuda()
   # a.device and b.device are device(type='cuda', index=1)

   # You can also use ``Tensor.to`` to transfer a tensor:
   b2 = torch.tensor([1., 2.]).to(device=cuda)
   # b.device and b2.device are device(type='cuda', index=1)

   c = a + b
   # c.device is device(type='cuda', index=1)

   z = x + y
   # z.device is device(type='cuda', index=0)

   # even within a context, you can specify the device
   # (or give a GPU index to the .cuda call)
   d = torch.randn(2, device=cuda2)
   e = torch.randn(2).to(cuda2)
   f = torch.randn(2).cuda(cuda2)