shrinath-suresh/blogs-docs-splitted huggingface dataset:
torch.nn.utils.rnn.pad_packed_sequence
torch.nn.utils.rnn.pad_packed_sequence(sequence, batch_first=False, padding_value=0.0, total_length=None)
Pads a packed batch of variable length sequences.
It is an inverse operation to "pack_padded_sequence()".
The returned Tensor's data will be of size "T x B x ", where T
   is the length of the longest sequence and B is the batch size. If
   "batch_first" is True, the data will be transposed into "B x T x "
   format.
-[ Example ]-



from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence
seq = torch.tensor([[1, 2, 0], [3, 0, 0], [4, 5, 6]])
lens = [2, 1, 3]
packed = pack_padded_sequence(seq, lens, batch_first=True, enforce_sorted=False)
packed
   PackedSequence(data=tensor([4, 1, 3, 5, 2, 6]), batch_sizes=tensor([3, 2, 1]),
                  sorted_indices=tensor([2, 0, 1]), unsorted_indices=tensor([1, 2, 0]))
seq_unpacked, lens_unpacked = pad_packed_sequence(packed, batch_first=True)
seq_unpacked
   tensor([[1, 2, 0],
           [3, 0, 0],
           [4, 5, 6]])
lens_unpacked
   tensor([2, 1, 3])



Note:
 "total_length" is useful to implement the "pack sequence ->
 recurrent network -> unpack sequence" pattern in a "Module"
 wrapped in "DataParallel". See this FAQ section for details.

Parameters:
      * sequence (PackedSequence) -- batch to pad
  * **batch_first** (*bool**, **optional*) -- if "True", the
    output will be in "B x T x *" format.

  * **padding_value** (*float**, **optional*) -- values for padded
    elements.

  * **total_length** (*int**, **optional*) -- if not "None", the
    output will be padded to have length "total_length". This
    method will throw "ValueError" if "total_length" is less than
    the max sequence length in "sequence".

Returns:
torch.chain_matmul
torch.chain_matmul(*matrices, out=None)
Returns the matrix product of the N 2-D tensors. This product is
   efficiently computed using the matrix chain order algorithm which
   selects the order in which incurs the lowest cost in terms of
   arithmetic operations ([CLRS]). Note that since this is a function
   to compute the product, N needs to be greater than or equal to 2;
   if equal to 2 then a trivial matrix-matrix product is returned. If
   N is 1, then this is a no-op - the original matrix is returned as
   is.
Warning:
 "torch.chain_matmul()" is deprecated and will be removed in a
 future PyTorch release. Use "torch.linalg.multi_dot()" instead,
 which accepts a list of two or more tensors rather than multiple
 arguments.

Parameters:
      * matrices (Tensors...) -- a sequence of 2 or more 2-D
        tensors whose product is to be determined.
  * **out** (*Tensor**, **optional*) -- the output tensor. Ignored
if "out" = "None".
Returns:
      if the i^{th} tensor was of dimensions p_{i} \times p_{i + 1},
      then the product would be of dimensions p_{1} \times p_{N + 1}.
Return type:
      Tensor
Example:
  >>> a = torch.randn(3, 4)
  >>> b = torch.randn(4, 5)
  >>> c = torch.randn(5, 6)
  >>> d = torch.randn(6, 7)
  >>> # will raise a deprecation warning
  >>> torch.chain_matmul(a, b, c, d)
  tensor([[ -2.3375,  -3.9790,  -4.1119,  -6.6577,   9.5609, -11.5095,  -3.2614],
          [ 21.4038,   3.3378,  -8.4982,  -5.2457, -10.2561,  -2.4684,   2.7163],
          [ -0.9647,  -5.8917,  -2.3213,  -5.2284,  12.8615, -12.2816,  -2.5095]])
torch.cuda.utilization
torch.cuda.utilization(device=None)
Returns the percent of time over the past sample period during
   which one or more kernels was executing on the GPU as given by
   nvidia-smi.
Parameters:
      device (torch.device or int, optional) -- selected
      device. Returns statistic for the current device, given by
      "current_device()", if "device" is "None" (default).
Return type:
      int
Warning: Each sample period may be between 1 second and 1/6 second,
   depending on the product being queried.
CrossEntropyLoss
class torch.nn.CrossEntropyLoss(weight=None, size_average=None, ignore_index=- 100, reduce=None, reduction='mean', label_smoothing=0.0)
This criterion computes the cross entropy loss between input logits
   and target.
It is useful when training a classification problem with C
   classes. If provided, the optional argument "weight" should be a 1D
   Tensor assigning weight to each of the classes. This is
   particularly useful when you have an unbalanced training set.
The input is expected to contain the unnormalized logits for each
   class (which do not need to be positive or sum to 1, in general).
   input has to be a Tensor of size (C) for unbatched input,
   (minibatch, C) or (minibatch, C, d_1, d_2, ..., d_K) with K \geq 1
   for the K-dimensional case. The last being useful for higher
   dimension inputs, such as computing cross entropy loss per-pixel
   for 2D images.
The target that this criterion expects should contain either:
Class indices in the range [0, C) where C is the number of
     classes; if ignore_index is specified, this loss also accepts
     this class index (this index may not necessarily be in the class
     range). The unreduced (i.e. with "reduction" set to "'none'")
     loss for this case can be described as:
\ell(x, y) = L = \{l_1,\dots,l_N\}^\top, \quad l_n = - w_{y_n}
\log \frac{\exp(x_{n,y_n})}{\sum_{c=1}^C \exp(x_{n,c})} \cdot
\mathbb{1}\{y_n \not= \text{ignore\_index}\}

where x is the input, y is the target, w is the weight, C is the
 number of classes, and N spans the minibatch dimension as well as
 d_1, ..., d_k for the K-dimensional case. If "reduction" is not
 "'none'" (default "'mean'"), then
\ell(x, y) = \begin{cases}     \sum_{n=1}^N
\frac{1}{\sum_{n=1}^N w_{y_n} \cdot \mathbb{1}\{y_n \not=
\text{ignore\_index}\}} l_n, &      \text{if reduction} =
\text{`mean';}\\       \sum_{n=1}^N l_n,  &       \text{if
reduction} = \text{`sum'.}   \end{cases}
 Note that this case is equivalent to the combination of
 "LogSoftmax" and "NLLLoss".



Probabilities for each class; useful when labels beyond a single
     class per minibatch item are required, such as for blended
     labels, label smoothing, etc. The unreduced (i.e. with
     "reduction" set to "'none'") loss for this case can be described
     as:
\ell(x, y) = L = \{l_1,\dots,l_N\}^\top, \quad l_n = -
\sum_{c=1}^C w_c \log \frac{\exp(x_{n,c})}{\sum_{i=1}^C
\exp(x_{n,i})} y_{n,c}

where x is the input, y is the target, w is the weight, C is the
 number of classes, and N spans the minibatch dimension as well as
 d_1, ..., d_k for the K-dimensional case. If "reduction" is not
 "'none'" (default "'mean'"), then
\ell(x, y) = \begin{cases}     \frac{\sum_{n=1}^N l_n}{N}, &
\text{if reduction} = \text{`mean';}\\       \sum_{n=1}^N l_n,
&       \text{if reduction} = \text{`sum'.}   \end{cases}
Note:
 The performance of this criterion is generally better when
 *target* contains class indices, as this allows for optimized
 computation. Consider providing *target* as class probabilities
 only when a single class label per minibatch item is too
 restrictive.

Parameters:
      * weight (Tensor, optional) -- a manual rescaling
        weight given to each class. If given, has to be a Tensor of
        size C
  * **size_average** (*bool**, **optional*) -- Deprecated (see
    "reduction"). By default, the losses are averaged over each
    loss element in the batch. Note that for some losses, there
    are multiple elements per sample. If the field "size_average"
    is set to "False", the losses are instead summed for each
    minibatch. Ignored when "reduce" is "False". Default: "True"

  * **ignore_index** (*int**, **optional*) -- Specifies a target
value that is ignored and does not contribute to the input
        gradient. When "size_average" is "True", the loss is averaged
        over non-ignored targets. Note that "ignore_index" is only
        applicable when the target contains class indices.
  * **reduce** (*bool**, **optional*) -- Deprecated (see
    "reduction"). By default, the losses are averaged or summed
    over observations for each minibatch depending on
    "size_average". When "reduce" is "False", returns a loss per
    batch element instead and ignores "size_average". Default:
    "True"

  * **reduction** (*str**, **optional*) -- Specifies the reduction
    to apply to the output: "'none'" | "'mean'" | "'sum'".
    "'none'": no reduction will be applied, "'mean'": the weighted
    mean of the output is taken, "'sum'": the output will be
    summed. Note: "size_average" and "reduce" are in the process
    of being deprecated, and in the meantime, specifying either of
those two args will override "reduction". Default: "'mean'"
  * **label_smoothing** (*float**, **optional*) -- A float in
    [0.0, 1.0]. Specifies the amount of smoothing when computing
    the loss, where 0.0 means no smoothing. The targets become a
    mixture of the original ground truth and a uniform
    distribution as described in Rethinking the Inception
    Architecture for Computer Vision. Default: 0.0.

Shape:
      * Input: Shape (C), (N, C) or (N, C, d_1, d_2, ..., d_K) with K
        \geq 1 in the case of K-dimensional loss.
  * Target: If containing class indices, shape (), (N) or (N, d_1,
    d_2, ..., d_K) with K \geq 1 in the case of K-dimensional loss
    where each value should be between [0, C). If containing class
    probabilities, same shape as the input and each value should
    be between [0, 1].

  * Output: If reduction is 'none', shape (), (N) or (N, d_1, d_2,
..., d_K) with K \geq 1 in the case of K-dimensional loss,
        depending on the shape of the input. Otherwise, scalar.
  where:

     \begin{aligned}     C ={} & \text{number of classes} \\     N
     ={} & \text{batch size} \\ \end{aligned}

Examples:
  >>> # Example of target with class indices
  >>> loss = nn.CrossEntropyLoss()
  >>> input = torch.randn(3, 5, requires_grad=True)
  >>> target = torch.empty(3, dtype=torch.long).random_(5)
  >>> output = loss(input, target)
  >>> output.backward()
  >>>
  >>> # Example of target with class probabilities
  >>> input = torch.randn(3, 5, requires_grad=True)
  >>> target = torch.randn(3, 5).softmax(dim=1)
  >>> output = loss(input, target)
  >>> output.backward()
prepare_qat
class torch.quantization.prepare_qat(model, mapping=None, inplace=False)
Prepares a copy of the model for quantization calibration or
   quantization-aware training and converts it to quantized version.
Quantization configuration should be assigned preemptively to
   individual submodules in .qconfig attribute.
Parameters:
      * model -- input model to be modified in-place
  * **mapping** -- dictionary that maps float modules to quantized
    modules to be replaced.

  * **inplace** -- carry out model transformations in-place, the
    original module is mutated
torch.sum
torch.sum(input, *, dtype=None) -> Tensor
Returns the sum of all elements in the "input" tensor.
Parameters:
      input (Tensor) -- the input tensor.
Keyword Arguments:
      dtype ("torch.dtype", optional) -- the desired data type of
      returned tensor. If specified, the input tensor is casted to
      "dtype" before the operation is performed. This is useful for
      preventing data type overflows. Default: None.
Example:
  >>> a = torch.randn(1, 3)
  >>> a
  tensor([[ 0.1133, -0.9567,  0.2958]])
  >>> torch.sum(a)
  tensor(-0.5475)

torch.sum(input, dim, keepdim=False, *, dtype=None) -> Tensor
Returns the sum of each row of the "input" tensor in the given
   dimension "dim". If "dim" is a list of dimensions, reduce over all
   of them.
If "keepdim" is "True", the output tensor is of the same size as
   "input" except in the dimension(s) "dim" where it is of size 1.
Otherwise, "dim" is squeezed (see "torch.squeeze()"), resulting in
   the output tensor having 1 (or "len(dim)") fewer dimension(s).
Parameters:
      * input (Tensor) -- the input tensor.
  * **dim** (*int** or **tuple of ints**, **optional*) -- the
    dimension or dimensions to reduce. If "None", all dimensions
    are reduced.

  * **keepdim** (*bool*) -- whether the output tensor has "dim"
    retained or not.

Keyword Arguments:
      dtype ("torch.dtype", optional) -- the desired data type of
      returned tensor. If specified, the input tensor is casted to
      "dtype" before the operation is performed. This is useful for
      preventing data type overflows. Default: None.
Example:
  >>> a = torch.randn(4, 4)
  >>> a
  tensor([[ 0.0569, -0.2475,  0.0737, -0.3429],
          [-0.2993,  0.9138,  0.9337, -1.6864],
          [ 0.1132,  0.7892, -0.1003,  0.5688],
          [ 0.3637, -0.9906, -0.4752, -1.5197]])
torch.sum(a, 1)
      tensor([-0.4598, -0.1381,  1.3708, -2.6217])
      >>> b = torch.arange(4 * 5 * 6).view(4, 5, 6)
      >>> torch.sum(b, (2, 1))
      tensor([  435.,  1335.,  2235.,  3135.])
torch.autograd.grad
torch.autograd.grad(outputs, inputs, grad_outputs=None, retain_graph=None, create_graph=False, only_inputs=True, allow_unused=False, is_grads_batched=False)
Computes and returns the sum of gradients of outputs with respect
   to the inputs.
"grad_outputs" should be a sequence of length matching "output"
   containing the "vector" in vector-Jacobian product, usually the
   pre-computed gradients w.r.t. each of the outputs. If an output
   doesn't require_grad, then the gradient can be "None").
Note:
 If you run any forward ops, create "grad_outputs", and/or call
 "grad" in a user-specified CUDA stream context, see Stream
 semantics of backward passes.

Note:
 "only_inputs" argument is deprecated and is ignored now (defaults
 to "True"). To accumulate gradient for other parts of the graph,
 please use "torch.autograd.backward".

Parameters:
      * outputs (sequence of Tensor) -- outputs of the
differentiated function.
  * **inputs** (*sequence of Tensor*) -- Inputs w.r.t. which the
    gradient will be returned (and not accumulated into ".grad").

  * **grad_outputs** (*sequence of Tensor*) -- The "vector" in the
    vector-Jacobian product. Usually gradients w.r.t. each output.
    None values can be specified for scalar Tensors or ones that
    don't require grad. If a None value would be acceptable for
    all grad_tensors, then this argument is optional. Default:
    None.

  * **retain_graph** (*bool**, **optional*) -- If "False", the
    graph used to compute the grad will be freed. Note that in
    nearly all cases setting this option to "True" is not needed
    and often can be worked around in a much more efficient way.
    Defaults to the value of "create_graph".

  * **create_graph** (*bool**, **optional*) -- If "True", graph of
    the derivative will be constructed, allowing to compute higher
order derivative products. Default: "False".
  * **allow_unused** (*bool**, **optional*) -- If "False",
    specifying inputs that were not used when computing outputs
    (and therefore their grad is always zero) is an error.
    Defaults to "False".

  * **is_grads_batched** (*bool**, **optional*) -- If "True", the
    first dimension of each tensor in "grad_outputs" will be
    interpreted as the batch dimension. Instead of computing a
    single vector-Jacobian product, we compute a batch of vector-
    Jacobian products for each "vector" in the batch. We use the
    vmap prototype feature as the backend to vectorize calls to
    the autograd engine so that this computation can be performed
    in a single call. This should lead to performance improvements
    when compared to manually looping and performing backward
    multiple times. Note that due to this feature being
    experimental, there may be performance cliffs. Please use
torch.Tensor.index_add_
Tensor.index_add_(dim, index, source, *, alpha=1) -> Tensor
Accumulate the elements of "alpha" times "source" into the "self"
   tensor by adding to the indices in the order given in "index". For
   example, if "dim == 0", "index[i] == j", and "alpha=-1", then the
   "i"th row of "source" is subtracted from the "j"th row of "self".
The "dim"th dimension of "source" must have the same size as the
   length of "index" (which must be a vector), and all other
   dimensions must match "self", or an error will be raised.
For a 3-D tensor the output is given as:
  self[index[i], :, :] += alpha * src[i, :, :]  # if dim == 0
  self[:, index[i], :] += alpha * src[:, i, :]  # if dim == 1
  self[:, :, index[i]] += alpha * src[:, :, i]  # if dim == 2

Note:
 This operation may behave nondeterministically when given tensors
 on a CUDA device. See Reproducibility for more information.

Parameters:
Parameters:
      * dim (int) -- dimension along which to index
  * **index** (*Tensor*) -- indices of "source" to select from,
    should have dtype either *torch.int64* or *torch.int32*

  * **source** (*Tensor*) -- the tensor containing values to add

Keyword Arguments:
      alpha (Number) -- the scalar multiplier for "source"
Example:
  >>> x = torch.ones(5, 3)
  >>> t = torch.tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]], dtype=torch.float)
  >>> index = torch.tensor([0, 4, 2])
  >>> x.index_add_(0, index, t)
  tensor([[  2.,   3.,   4.],
          [  1.,   1.,   1.],
          [  8.,   9.,  10.],
          [  1.,   1.,   1.],
          [  5.,   6.,   7.]])
  >>> x.index_add_(0, index, t, alpha=-1)
  tensor([[  1.,   1.,   1.],
          [  1.,   1.,   1.],
          [  1.,   1.,   1.],
          [  1.,   1.,   1.],
          [  1.,   1.,   1.]])
torch.Tensor.bfloat16
Tensor.bfloat16(memory_format=torch.preserve_format) -> Tensor
"self.bfloat16()" is equivalent to "self.to(torch.bfloat16)". See
   "to()".
Parameters:
      memory_format ("torch.memory_format", optional) -- the
      desired memory format of returned Tensor. Default:
      "torch.preserve_format".
torch.tensordot
torch.tensordot(a, b, dims=2, out=None)
Returns a contraction of a and b over multiple dimensions.
"tensordot" implements a generalized matrix product.
Parameters:
      * a (Tensor) -- Left tensor to contract
  * **b** (*Tensor*) -- Right tensor to contract

  * **dims** (*int** or **Tuple**[**List**[**int**]**,
    **List**[**int**]**] or **List**[**List**[**int**]**]
    **containing two lists** or **Tensor*) -- number of dimensions
    to contract or explicit lists of dimensions for "a" and "b"
    respectively

When called with a non-negative integer argument "dims" = d, and
   the number of dimensions of "a" and "b" is m and n, respectively,
   "tensordot()" computes
  r_{i_0,...,i_{m-d}, i_d,...,i_n}   = \sum_{k_0,...,k_{d-1}}
  a_{i_0,...,i_{m-d},k_0,...,k_{d-1}} \times b_{k_0,...,k_{d-1},
  i_d,...,i_n}.

When called with "dims" of the list form, the given dimensions will
be contracted in place of the last d of "a" and the first d of b.
   The sizes in these dimensions must match, but "tensordot()" will
   deal with broadcasted dimensions.
Examples:
  >>> a = torch.arange(60.).reshape(3, 4, 5)
  >>> b = torch.arange(24.).reshape(4, 3, 2)
  >>> torch.tensordot(a, b, dims=([1, 0], [0, 1]))
  tensor([[4400., 4730.],
          [4532., 4874.],
          [4664., 5018.],
          [4796., 5162.],
          [4928., 5306.]])

  >>> a = torch.randn(3, 4, 5, device='cuda')
  >>> b = torch.randn(4, 5, 6, device='cuda')
  >>> c = torch.tensordot(a, b, dims=2).cpu()
  tensor([[ 8.3504, -2.5436,  6.2922,  2.7556, -1.0732,  3.2741],
          [ 3.3161,  0.0704,  5.0187, -0.4079, -4.3126,  4.8744],
          [ 0.8223,  3.9445,  3.2168, -0.2400,  3.4117,  1.7780]])

  >>> a = torch.randn(3, 5, 4, 6)
  >>> b = torch.randn(6, 4, 5, 3)
  >>> torch.tensordot(a, b, dims=([2, 1, 3], [1, 2, 0]))
torch.signal.windows.nuttall
torch.signal.windows.nuttall(M, *, sym=True, dtype=None, layout=torch.strided, device=None, requires_grad=False)
Computes the minimum 4-term Blackman-Harris window according to
   Nuttall.
  w_n = 1 - 0.36358 \cos{(z_n)} + 0.48917 \cos{(2z_n)} - 0.13659
  \cos{(3z_n)} + 0.01064 \cos{(4z_n)}

where "z_n = 2 Ï n/ M".
The window is normalized to 1 (maximum value is 1). However, the 1
   doesn't appear if "M" is even and "sym" is True.
Parameters:
      M (int) -- the length of the window. In other words, the
      number of points of the returned window.
Keyword Arguments:
      * sym (bool, optional) -- If False, returns a
        periodic window suitable for use in spectral analysis. If
        True, returns a symmetric window suitable for use in filter
        design. Default: True.
  * **dtype** ("torch.dtype", optional) -- the desired data type
of returned tensor. Default: if "None", uses a global default
        (see "torch.set_default_tensor_type()").
  * **layout** ("torch.layout", optional) -- the desired layout of
    returned Tensor. Default: "torch.strided".

  * **device** ("torch.device", optional) -- the desired device of
    returned tensor. Default: if "None", uses the current device
    for the default tensor type (see
    "torch.set_default_tensor_type()"). "device" will be the CPU
    for CPU tensor types and the current CUDA device for CUDA
    tensor types.

  * **requires_grad** (*bool**, **optional*) -- If autograd should
    record operations on the returned tensor. Default: "False".

Return type:
      Tensor
References:
  - A. Nuttall, âSome windows with very good sidelobe behavior,â
    IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 29, no. 1, pp. 84-91,
    Feb 1981. https://doi.org/10.1109/TASSP.1981.1163506
Heinzel G. et al., âSpectrum and spectral density estimation by the Discrete Fourier transform (DFT),
        including a comprehensive list of window functions and some new flat-top windowsâ,
        February 15, 2002 https://holometer.fnal.gov/GH_FFT.pdf

Examples:
  >>> # Generates a symmetric Nutall window.
  >>> torch.signal.windows.general_hamming(5, sym=True)
  tensor([3.6280e-04, 2.2698e-01, 1.0000e+00, 2.2698e-01, 3.6280e-04])

  >>> # Generates a periodic Nuttall window.
  >>> torch.signal.windows.general_hamming(5, sym=False)
  tensor([3.6280e-04, 1.1052e-01, 7.9826e-01, 7.9826e-01, 1.1052e-01])
Upsample
class torch.nn.Upsample(size=None, scale_factor=None, mode='nearest', align_corners=None, recompute_scale_factor=None)
Upsamples a given multi-channel 1D (temporal), 2D (spatial) or 3D
   (volumetric) data.
The input data is assumed to be of the form minibatch x channels x
   [optional depth] x [optional height] x width. Hence, for spatial
   inputs, we expect a 4D Tensor and for volumetric inputs, we expect
   a 5D Tensor.
The algorithms available for upsampling are nearest neighbor and
   linear, bilinear, bicubic and trilinear for 3D, 4D and 5D input
   Tensor, respectively.
One can either give a "scale_factor" or the target output "size" to
   calculate the output size. (You cannot give both, as it is
   ambiguous)
Parameters:
      * size (int or Tuple[int] or Tuple[int,
        int] or Tuple[int, int, int],
        optional) -- output spatial sizes
  * **scale_factor** (*float** or **Tuple**[**float**] or
Tuple[float, float] or Tuple[float,
        float, float], optional*) -- multiplier for
        spatial size. Has to match input size if it is a tuple.
  * **mode** (*str**, **optional*) -- the upsampling algorithm:
    one of "'nearest'", "'linear'", "'bilinear'", "'bicubic'" and
    "'trilinear'". Default: "'nearest'"

  * **align_corners** (*bool**, **optional*) -- if "True", the
    corner pixels of the input and output tensors are aligned, and
    thus preserving the values at those pixels. This only has
    effect when "mode" is "'linear'", "'bilinear'", "'bicubic'",
    or "'trilinear'". Default: "False"

  * **recompute_scale_factor** (*bool**, **optional*) -- recompute
    the scale_factor for use in the interpolation calculation. If
    *recompute_scale_factor* is "True", then *scale_factor* must
    be passed in and *scale_factor* is used to compute the output