shrinath-suresh/blogs-docs-splitted huggingface dataset:
with ddp.no_sync():
         >>>     for input in inputs:
         >>>         ddp(input).backward()  # no synchronization, accumulate grads
         >>> ddp(another_input).backward()  # synchronize grads



  Warning:

    The forward pass should be included inside the context
    manager, or else gradients will still be synchronized.

register_comm_hook(state, hook)
  Registers a communication hook which is an enhancement that
  provides a flexible hook to users where they can specify how DDP
  aggregates gradients across multiple workers.

  This hook would be very useful for researchers to try out new
  ideas. For example, this hook can be used to implement several
  algorithms like GossipGrad and gradient compression which
  involve different communication strategies for parameter syncs
  while running Distributed DataParallel training.

  Parameters:
     * **state** (*object*) --
state (object) --   Passed to the hook to maintain any state information during
   the training process. Examples include error feedback in
   gradient compression, peers to communicate with next in
   GossipGrad, etc.

   It is locally stored by each worker and shared by all the
   gradient tensors on the worker.

 * **hook** (*Callable*) --

   Callable with the following signature: "hook(state: object,
   bucket: dist.GradBucket) ->
   torch.futures.Future[torch.Tensor]":

   This function is called once the bucket is ready. The hook
   can perform whatever processing is needed and return a
   Future indicating completion of any async work (ex:
   allreduce). If the hook doesn't perform any communication,
   it still must return a completed Future. The Future should
   hold the new value of grad bucket's tensors. Once a bucket
is ready, c10d reducer would call this hook and use the
           tensors returned by the Future and copy grads to individual
           parameters. Note that the future's return type must be a
           single tensor.
       We also provide an API called "get_future" to retrieve a
       Future associated with the completion of
       "c10d.ProcessGroup.Work". "get_future" is currently
       supported for NCCL and also supported for most operations
       on GLOO and MPI, except for peer to peer operations
       (send/recv).

  Warning:

    Grad bucket's tensors will not be predivided by world_size.
    User is responsible to divide by the world_size in case of
    operations like allreduce.

  Warning:

    DDP communication hook can only be registered once and should
    be registered before calling backward.

  Warning:

    The Future object that hook returns should contain a single
tensor that has the same shape with the tensors inside grad
        bucket.
  Warning:

    "get_future" API supports NCCL, and partially GLOO and MPI
    backends (no support for peer-to-peer operations like
    send/recv) and will return a "torch.futures.Future".

  Example::
     Below is an example of a noop hook that returns the same
     tensor.

     >>> def noop(state: object, bucket: dist.GradBucket) -> torch.futures.Future[torch.Tensor]:
     >>>     fut = torch.futures.Future()
     >>>     fut.set_result(bucket.buffer())
     >>>     return fut
     >>> ddp.register_comm_hook(state=None, hook=noop)

  Example::
     Below is an example of a Parallel SGD algorithm where
     gradients are encoded before allreduce, and then decoded
     after allreduce.

     >>> def encode_and_decode(state: object, bucket: dist.GradBucket) -> torch.futures.Future[torch.Tensor]:
encoded_tensor = encode(bucket.buffer())  # encode gradients
     >>>     fut = torch.distributed.all_reduce(encoded_tensor).get_future()
     >>>     # Define the then callback to decode.
     >>>     def decode(fut):
     >>>         decoded_tensor = decode(fut.value()[0])  # decode gradients
     >>>         return decoded_tensor
     >>>     return fut.then(decode)
     >>> ddp.register_comm_hook(state=None, hook=encode_and_decode)
torch.sparse_compressed_tensor
torch.sparse_compressed_tensor(compressed_indices, plain_indices, values, size=None, *, dtype=None, layout=None, device=None, requires_grad=False, check_invariants=None) -> Tensor
Constructs a sparse tensor in Compressed Sparse format - CSR, CSC,
   BSR, or BSC - with specified values at the given
   "compressed_indices" and "plain_indices". Sparse matrix
   multiplication operations in Compressed Sparse format are typically
   faster than that for sparse tensors in COO format. Make you have a
   look at the note on the data type of the indices.
Note:
 If the "device" argument is not specified the device of the given
 "values" and indices tensor(s) must match. If, however, the
 argument is specified the input Tensors will be converted to the
 given device and in turn determine the device of the constructed
 sparse tensor.

Parameters:
      * compressed_indices (array_like) -- (B+1)-dimensional
array of size "(*batchsize, compressed_dim_size + 1)".  The
        last element of each batch is the number of non-zero elements
        or blocks. This tensor encodes the index in "values" and
        "plain_indices" depending on where the given compressed
        dimension (row or column) starts. Each successive number in
        the tensor subtracted by the number before it denotes the
        number of elements or blocks in a given compressed dimension.
  * **plain_indices** (*array_like*) -- Plain dimension (column or
    row) co-ordinates of each element or block in values.
    (B+1)-dimensional tensor with the same length as values.

  * **values** (*array_list*) -- Initial values for the tensor.
    Can be a list, tuple, NumPy "ndarray", scalar, and other
    types.  that represents a (1+K)-dimensional (for CSR and CSC
    layouts) or (1+2+K)-dimensional tensor (for BSR and BSC
    layouts) where "K" is the number of dense dimensions.
size (list, tuple, "torch.Size", optional) -- Size of the
        sparse tensor: "(batchsize, nrows * blocksize[0], ncols *
        blocksize[1], densesize)" where "blocksize[0] == blocksize[1]
        == 1" for CSR and CSC formats. If not provided, the size will
        be inferred as the minimum size big enough to hold all non-
        zero elements or blocks.

Keyword Arguments:
      * dtype ("torch.dtype", optional) -- the desired data type
        of returned tensor.  Default: if None, infers data type from
        "values".
  * **layout** ("torch.layout", required) -- the desired layout of
    returned tensor: "torch.sparse_csr", "torch.sparse_csc",
    "torch.sparse_bsr", or "torch.sparse_bsc".

  * **device** ("torch.device", optional) -- the desired device of
    returned tensor.  Default: if None, uses the current device
    for the default tensor type (see
    "torch.set_default_tensor_type()"). "device" will be the CPU
for CPU tensor types and the current CUDA device for CUDA
        tensor types.
  * **requires_grad** (*bool**, **optional*) -- If autograd should
    record operations on the returned tensor. Default: "False".

  * **check_invariants** (*bool**, **optional*) -- If sparse
    tensor invariants are checked. Default: as returned by
    "torch.sparse.check_sparse_tensor_invariants.is_enabled()",
    initially False.

Example::
      >>> compressed_indices = [0, 2, 4]
      >>> plain_indices = [0, 1, 0, 1]
      >>> values = [1, 2, 3, 4]
      >>> torch.sparse_compressed_tensor(torch.tensor(compressed_indices, dtype=torch.int64),
      ...                                torch.tensor(plain_indices, dtype=torch.int64),
      ...                                torch.tensor(values), dtype=torch.double, layout=torch.sparse_csr)
      tensor(crow_indices=tensor([0, 2, 4]),
             col_indices=tensor([0, 1, 0, 1]),
             values=tensor([1., 2., 3., 4.]), size=(2, 2), nnz=4,
InstanceNorm3d
class torch.nn.InstanceNorm3d(num_features, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False, device=None, dtype=None)
Applies Instance Normalization over a 5D input (a mini-batch of 3D
   inputs with additional channel dimension) as described in the paper
   Instance Normalization: The Missing Ingredient for Fast
   Stylization.
  y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}}
  * \gamma + \beta

The mean and standard-deviation are calculated per-dimension
   separately for each object in a mini-batch. \gamma and \beta are
   learnable parameter vectors of size C (where C is the input size)
   if "affine" is "True". The standard-deviation is calculated via the
   biased estimator, equivalent to torch.var(input, unbiased=False).
By default, this layer uses instance statistics computed from input
   data in both training and evaluation modes.
If "track_running_stats" is set to "True", during training this
layer keeps running estimates of its computed mean and variance,
   which are then used for normalization during evaluation. The
   running estimates are kept with a default "momentum" of 0.1.
Note:
 This "momentum" argument is different from one used in optimizer
 classes and the conventional notion of momentum. Mathematically,
 the update rule for running statistics here is \hat{x}_\text{new}
 = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times
 x_t, where \hat{x} is the estimated statistic and x_t is the new
 observed value.

Note:
 "InstanceNorm3d" and "LayerNorm" are very similar, but have some
 subtle differences. "InstanceNorm3d" is applied on each channel
 of channeled data like 3D models with RGB color, but "LayerNorm"
 is usually applied on entire sample and often in NLP tasks.
 Additionally, "LayerNorm" applies elementwise affine transform,
 while "InstanceNorm3d" usually don't apply affine transform.

Parameters:
Parameters:
      * num_features (int) -- C from an expected input of size
        (N, C, D, H, W) or (C, D, H, W)
  * **eps** (*float*) -- a value added to the denominator for
    numerical stability. Default: 1e-5

  * **momentum** (*float*) -- the value used for the running_mean
    and running_var computation. Default: 0.1

  * **affine** (*bool*) -- a boolean value that when set to
    "True", this module has learnable affine parameters,
    initialized the same way as done for batch normalization.
    Default: "False".

  * **track_running_stats** (*bool*) -- a boolean value that when
    set to "True", this module tracks the running mean and
    variance, and when set to "False", this module does not track
    such statistics and always uses batch statistics in both
    training and eval modes. Default: "False"

Shape:
      * Input: (N, C, D, H, W) or (C, D, H, W)
  * Output: (N, C, D, H, W) or (C, D, H, W) (same shape as input)
torch.nn.functional.bilinear
torch.nn.functional.bilinear(input1, input2, weight, bias=None) -> Tensor
Applies a bilinear transformation to the incoming data: y = x_1^T A
   x_2 + b
Shape:
  * input1: (N, *, H_{in1}) where H_{in1}=\text{in1\_features} and
    * means any number of additional dimensions. All but the last
    dimension of the inputs should be the same.

  * input2: (N, *, H_{in2}) where H_{in2}=\text{in2\_features}

  * weight: (\text{out\_features}, \text{in1\_features},
    \text{in2\_features})

  * bias: (\text{out\_features})

  * output: (N, *, H_{out}) where H_{out}=\text{out\_features} and
    all but the last dimension are the same shape as the input.
torch.linalg.householder_product
torch.linalg.householder_product(A, tau, *, out=None) -> Tensor
Computes the first n columns of a product of Householder
   matrices.
Let \mathbb{K} be \mathbb{R} or \mathbb{C}, and let V \in
   \mathbb{K}^{m \times n} be a matrix with columns v_i \in
   \mathbb{K}^m for i=1,\ldots,m with m \geq n. Denote by w_i the
   vector resulting from zeroing out the first i-1 components of v_i
   and setting to 1 the i-th. For a vector \tau \in \mathbb{K}^k
   with k \leq n, this function computes the first n columns of the
   matrix
  H_1H_2 ... H_k \qquad\text{with}\qquad H_i = \mathrm{I}_m -
  \tau_i w_i w_i^{\text{H}}

where \mathrm{I}_m is the m-dimensional identity matrix and
   w^{\text{H}} is the conjugate transpose when w is complex, and the
   transpose when w is real-valued. The output matrix is the same size
   as the input matrix "A".
See Representation of Orthogonal or Unitary Matrices for further
   details.
details.
Supports inputs of float, double, cfloat and cdouble dtypes. Also
   supports batches of matrices, and if the inputs are batches of
   matrices then the output has the same batch dimensions.
See also:
 "torch.geqrf()" can be used together with this function to form
 the *Q* from the "qr()" decomposition.

 "torch.ormqr()" is a related function that computes the matrix
 multiplication of a product of Householder matrices with another
 matrix. However, that function is not supported by autograd.

Warning:
 Gradient computations are only well-defined if tau_i \neq
 \frac{1}{||v_i||^2}. If this condition is not met, no error will
 be thrown, but the gradient produced may contain *NaN*.

Parameters:
      * A (Tensor) -- tensor of shape (, m, n)* where *** is
        zero or more batch dimensions.
  * **tau** (*Tensor*) -- tensor of shape *(*, k)* where *** is
    zero or more batch dimensions.

Keyword Arguments:
Keyword Arguments:
      out (Tensor, optional) -- output tensor. Ignored if
      None. Default: None.
Raises:
      RuntimeError -- if "A" doesn't satisfy the requirement m >=
      n,     or "tau" doesn't satisfy the requirement n >= k.
Examples:
  >>> A = torch.randn(2, 2)
  >>> h, tau = torch.geqrf(A)
  >>> Q = torch.linalg.householder_product(h, tau)
  >>> torch.dist(Q, torch.linalg.qr(A).Q)
  tensor(0.)

  >>> h = torch.randn(3, 2, 2, dtype=torch.complex128)
  >>> tau = torch.randn(3, 1, dtype=torch.complex128)
  >>> Q = torch.linalg.householder_product(h, tau)
  >>> Q
  tensor([[[ 1.8034+0.4184j,  0.2588-1.0174j],
          [-0.6853+0.7953j,  2.0790+0.5620j]],

          [[ 1.4581+1.6989j, -1.5360+0.1193j],
          [ 1.3877-0.6691j,  1.3512+1.3024j]],

          [[ 1.4766+0.5783j,  0.0361+0.6587j],
          [ 0.6396+0.1612j,  1.3693+0.4481j]]], dtype=torch.complex128)
torch.stack
torch.stack(tensors, dim=0, *, out=None) -> Tensor
Concatenates a sequence of tensors along a new dimension.
All tensors need to be of the same size.
Parameters:
      * tensors (sequence of Tensors) -- sequence of tensors to
        concatenate
  * **dim** (*int*) -- dimension to insert. Has to be between 0
    and the number of dimensions of concatenated tensors
    (inclusive)

Keyword Arguments:
      out (Tensor, optional) -- the output tensor.
torch.nextafter
torch.nextafter(input, other, *, out=None) -> Tensor
Return the next floating-point value after "input" towards "other",
   elementwise.
The shapes of "input" and "other" must be broadcastable.
Parameters:
      * input (Tensor) -- the first input tensor
  * **other** (*Tensor*) -- the second input tensor

Keyword Arguments:
      out (Tensor, optional) -- the output tensor.
Example:
  >>> eps = torch.finfo(torch.float32).eps
  >>> torch.nextafter(torch.tensor([1.0, 2.0]), torch.tensor([2.0, 1.0])) == torch.tensor([eps + 1, 2 - eps])
  tensor([True, True])
torch.fft.irfft2
torch.fft.irfft2(input, s=None, dim=(- 2, - 1), norm=None, *, out=None) -> Tensor
Computes the inverse of "rfft2()". Equivalent to "irfftn()" but
   IFFTs only the last two dimensions by default.
"input" is interpreted as a one-sided Hermitian signal in the
   Fourier domain, as produced by "rfft2()". By the Hermitian
   property, the output will be real-valued.
Note:
 Some input frequencies must be real-valued to satisfy the
 Hermitian property. In these cases the imaginary component will
 be ignored. For example, any imaginary component in the zero-
 frequency term cannot be represented in a real output and so will
 always be ignored.

Note:
 The correct interpretation of the Hermitian input depends on the
 length of the original data, as given by "s". This is because
 each input shape could correspond to either an odd or even length
 signal. By default, the signal is assumed to be even length and
odd signals will not round-trip properly. So, it is recommended
     to always pass the signal shape "s".
Note:
 Supports torch.half and torch.chalf on CUDA with GPU Architecture
 SM53 or greater. However it only supports powers of 2 signal
 length in every transformed dimensions. With default arguments,
 the size of last dimension should be (2^n + 1) as argument *s*
 defaults to even output size = 2 * (last_dim_size - 1)

Parameters:
      * input (Tensor) -- the input tensor
  * **s** (*Tuple**[**int**]**, **optional*) -- Signal size in the
    transformed dimensions. If given, each dimension "dim[i]" will
    either be zero-padded or trimmed to the length "s[i]" before
    computing the real FFT. If a length "-1" is specified, no
    padding is done in that dimension. Defaults to even output in
    the last dimension: "s[-1] = 2*(input.size(dim[-1]) - 1)".

  * **dim** (*Tuple**[**int**]**, **optional*) -- Dimensions to be
transformed. The last dimension must be the half-Hermitian
        compressed dimension. Default: last two dimensions.
  * **norm** (*str**, **optional*) --

    Normalization mode. For the backward transform ("irfft2()"),
    these correspond to:

    * ""forward"" - no normalization

    * ""backward"" - normalize by "1/n"

    * ""ortho"" - normalize by "1/sqrt(n)" (making the real IFFT
      orthonormal)

    Where "n = prod(s)" is the logical IFFT size. Calling the
    forward transform ("rfft2()") with the same normalization mode
    will apply an overall normalization of "1/n" between the two
    transforms. This is required to make "irfft2()" the exact
    inverse.

    Default is ""backward"" (normalize by "1/n").

Keyword Arguments:
      out (Tensor, optional) -- the output tensor.
-[ Example ]-



t = torch.rand(10, 9)
T = torch.fft.rfft2(t)
T = torch.fft.rfft2(t)



Without specifying the output length to "irfft2()", the output will
   not round-trip properly because the input is odd-length in the last
   dimension:



torch.fft.irfft2(T).size()
   torch.Size([10, 8])



So, it is recommended to always pass the signal shape "s".



roundtrip = torch.fft.irfft2(T, t.size())
roundtrip.size()
   torch.Size([10, 9])
torch.testing.assert_close(roundtrip, t, check_stride=False)
torch.var_mean
torch.var_mean(input, dim=None, *, correction=1, keepdim=False, out=None)
Calculates the variance and mean over the dimensions specified by
   "dim". "dim" can be a single dimension, list of dimensions, or
   "None" to reduce over all dimensions.
The variance (\sigma^2) is calculated as
  \sigma^2 = \frac{1}{N - \delta N}\sum_{i=0}^{N-1}(x_i-\bar{x})^2

where x is the sample set of elements, \bar{x} is the sample mean,
   N is the number of samples and \delta N is the "correction".
If "keepdim" is "True", the output tensor is of the same size as
   "input" except in the dimension(s) "dim" where it is of size 1.
   Otherwise, "dim" is squeezed (see "torch.squeeze()"), resulting in
   the output tensor having 1 (or "len(dim)") fewer dimension(s).
Parameters:
      * input (Tensor) -- the input tensor.
  * **dim** (*int** or **tuple of ints**, **optional*) -- the
    dimension or dimensions to reduce. If "None", all dimensions
are reduced.
Keyword Arguments:
      * correction (int) --
    difference between the sample size and sample degrees of
    freedom. Defaults to Bessel's correction, "correction=1".

    Changed in version 2.0: Previously this argument was called
    "unbiased" and was a boolean with "True" corresponding to
    "correction=1" and "False" being "correction=0".

  * **keepdim** (*bool*) -- whether the output tensor has "dim"
    retained or not.

  * **out** (*Tensor**, **optional*) -- the output tensor.

Returns:
      A tuple (var, mean) containing the variance and mean.
-[ Example ]-



a = torch.tensor(
   ...     [[ 0.2035,  1.2959,  1.8101, -0.4644],
   ...      [ 1.5027, -0.3270,  0.5905,  0.6538],
   ...      [-1.5745,  1.3330, -0.5596, -0.6548],
   ...      [ 0.1264, -0.5080,  1.6420,  0.1992]])
torch.var_mean(a, dim=0, keepdim=True)
   (tensor([[1.5926, 1.0056, 1.2005, 0.3646]]),
    tensor([[ 0.0645,  0.4485,  0.8707, -0.0665]]))
torch.Tensor.resize_
Tensor.resize_(*sizes, memory_format=torch.contiguous_format) -> Tensor
Resizes "self" tensor to the specified size. If the number of
   elements is larger than the current storage size, then the
   underlying storage is resized to fit the new number of elements. If
   the number of elements is smaller, the underlying storage is not
   changed. Existing elements are preserved but any new memory is
   uninitialized.
Warning:
 This is a low-level method. The storage is reinterpreted as
 C-contiguous, ignoring the current strides (unless the target
 size equals the current size, in which case the tensor is left
 unchanged). For most purposes, you will instead want to use
 "view()", which checks for contiguity, or "reshape()", which
 copies data if needed. To change the size in-place with custom
 strides, see "set_()".

Parameters:
      * sizes (torch.Size or int...) -- the desired size
memory_format ("torch.memory_format", optional) -- the
        desired memory format of Tensor. Default:
        "torch.contiguous_format". Note that memory format of "self"
        is going to be unaffected if "self.size()" matches "sizes".

Example:
  >>> x = torch.tensor([[1, 2], [3, 4], [5, 6]])
  >>> x.resize_(2, 2)
  tensor([[ 1,  2],
          [ 3,  4]])
torch.nn.functional.lp_pool2d
torch.nn.functional.lp_pool2d(input, norm_type, kernel_size, stride=None, ceil_mode=False)
Applies a 2D power-average pooling over an input signal composed of
   several input planes. If the sum of all inputs to the power of p
   is zero, the gradient is set to zero as well.
See "LPPool2d" for details.
Return type:
      Tensor
torch.sparse_bsr_tensor
torch.sparse_bsr_tensor(crow_indices, col_indices, values, size=None, *, dtype=None, device=None, requires_grad=False, check_invariants=None) -> Tensor
Constructs a sparse tensor in BSR (Block Compressed Sparse Row))
   with specified 2-dimensional blocks at the given "crow_indices" and
   "col_indices". Sparse matrix multiplication operations in BSR
   format are typically faster than that for sparse tensors in COO
   format. Make you have a look at the note on the data type of the
   indices.
Note:
 If the "device" argument is not specified the device of the given
 "values" and indices tensor(s) must match. If, however, the
 argument is specified the input Tensors will be converted to the
 given device and in turn determine the device of the constructed
 sparse tensor.

Parameters:
      * crow_indices (array_like) -- (B+1)-dimensional array of
        size "(*batchsize, nrowblocks + 1)".  The last element of each
batch is the number of non-zeros. This tensor encodes the
        block index in values and col_indices depending on where the
        given row block starts. Each successive number in the tensor
        subtracted by the number before it denotes the number of
        blocks in a given row.
  * **col_indices** (*array_like*) -- Column block co-ordinates of
    each block in values. (B+1)-dimensional tensor with the same
    length as values.

  * **values** (*array_list*) -- Initial values for the tensor.
    Can be a list, tuple, NumPy "ndarray", scalar, and other types
    that represents a (1 + 2 + K)-dimensional tensor where "K" is
    the number of dense dimensions.

  * **size** (list, tuple, "torch.Size", optional) -- Size of the
    sparse tensor: "(*batchsize, nrows * blocksize[0], ncols *
    blocksize[1], *densesize)" where "blocksize ==
    values.shape[1:3]". If not provided, the size will be inferred