shrinath-suresh/blogs-docs-splitted huggingface dataset:
output is given as:
  self[index[i], :, :] *= src[i, :, :]  # if dim == 0
  self[:, index[i], :] *= src[:, i, :]  # if dim == 1
  self[:, :, index[i]] *= src[:, :, i]  # if dim == 2

Note:
 This operation may behave nondeterministically when given tensors
 on a CUDA device. See Reproducibility for more information.

Note:
 This function only supports floating point tensors.

Warning:
 This function is in beta and may change in the near future.

Parameters:
      * dim (int) -- dimension along which to index
  * **index** (*Tensor*) -- indices of "source" to select from,
    should have dtype either *torch.int64* or *torch.int32*

  * **source** (*FloatTensor*) -- the tensor containing values to
    accumulate

  * **reduce** (*str*) -- the reduction operation to apply
    (""prod"", ""mean"", ""amax"", ""amin"")

Keyword Arguments:
      include_self (bool) -- whether the elements from the
"self" tensor are included in the reduction
Example:
  >>> x = torch.empty(5, 3).fill_(2)
  >>> t = torch.tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12]], dtype=torch.float)
  >>> index = torch.tensor([0, 4, 2, 0])
  >>> x.index_reduce_(0, index, t, 'prod')
  tensor([[20., 44., 72.],
          [ 2.,  2.,  2.],
          [14., 16., 18.],
          [ 2.,  2.,  2.],
          [ 8., 10., 12.]])
  >>> x = torch.empty(5, 3).fill_(2)
  >>> x.index_reduce_(0, index, t, 'prod', include_self=False)
  tensor([[10., 22., 36.],
          [ 2.,  2.,  2.],
          [ 7.,  8.,  9.],
          [ 2.,  2.,  2.],
          [ 4.,  5.,  6.]])
torch.autograd.functional.vhp
torch.autograd.functional.vhp(func, inputs, v=None, create_graph=False, strict=False)
Function that computes the dot product between a vector "v" and the
   Hessian of a given scalar function at the point given by the
   inputs.
Parameters:
      * func (function) -- a Python function that takes Tensor
        inputs and returns a Tensor with a single element.
  * **inputs** (*tuple of Tensors** or **Tensor*) -- inputs to the
    function "func".

  * **v** (*tuple of Tensors** or **Tensor*) -- The vector for
    which the vector Hessian product is computed. Must be the same
    size as the input of "func". This argument is optional when
    "func"'s input contains a single element and (if it is not
    provided) will be set as a Tensor containing a single "1".

  * **create_graph** (*bool**, **optional*) -- If "True", both the
    output and result will be computed in a differentiable way.
Note that when "strict" is "False", the result can not require
        gradients or be disconnected from the inputs. Defaults to
        "False".
  * **strict** (*bool**, **optional*) -- If "True", an error will
    be raised when we detect that there exists an input such that
    all the outputs are independent of it. If "False", we return a
    Tensor of zeros as the vhp for said inputs, which is the
    expected mathematical value. Defaults to "False".

Returns:
      tuple with:
         func_output (tuple of Tensors or Tensor): output of
         "func(inputs)"
     vhp (tuple of Tensors or Tensor): result of the dot product
     with the same shape as the inputs.

Return type:
      output (tuple)
-[ Example ]-



def pow_reducer(x):
   ...     return x.pow(3).sum()
inputs = torch.rand(2, 2)
v = torch.ones(2, 2)
vhp(pow_reducer, inputs, v)
   (tensor(0.5591),
    tensor([[1.0689, 1.2431],
            [3.0989, 4.4456]]))
[3.0989, 4.4456]]))



vhp(pow_reducer, inputs, v, create_graph=True)
   (tensor(0.5591, grad_fn=),
    tensor([[1.0689, 1.2431],
            [3.0989, 4.4456]], grad_fn=))
def pow_adder_reducer(x, y):
   ...     return (2 * x.pow(2) + 3 * y.pow(2)).sum()
inputs = (torch.rand(2), torch.rand(2))
v = (torch.zeros(2), torch.ones(2))
vhp(pow_adder_reducer, inputs, v)
   (tensor(4.8053),
    (tensor([0., 0.]),
     tensor([6., 6.])))
torch.nn.utils.prune.ln_structured
torch.nn.utils.prune.ln_structured(module, name, amount, n, dim, importance_scores=None)
Prunes tensor corresponding to parameter called "name" in "module"
   by removing the specified "amount" of (currently unpruned) channels
   along the specified "dim" with the lowest L"n"-norm. Modifies
   module in place (and also return the modified module) by:


adding a named buffer called "name+'_mask'" corresponding to the
      binary mask applied to the parameter "name" by the pruning
      method.


replacing the parameter "name" by its pruned version, while the
      original (unpruned) parameter is stored in a new parameter named
      "name+'_orig'".


Parameters:
      * module (nn.Module) -- module containing the tensor to
        prune
  * **name** (*str*) -- parameter name within "module" on which
    pruning will act.

  * **amount** (*int** or **float*) -- quantity of parameters to
prune. If "float", should be between 0.0 and 1.0 and represent
        the fraction of parameters to prune. If "int", it represents
        the absolute number of parameters to prune.
  * **n** (*int**, **float**, **inf**, **-inf**, **'fro'**,
    **'nuc'*) -- See documentation of valid entries for argument
    "p" in "torch.norm()".

  * **dim** (*int*) -- index of the dim along which we define
    channels to prune.

  * **importance_scores** (*torch.Tensor*) -- tensor of importance
    scores (of same shape as module parameter) used to compute
    mask for pruning. The values in this tensor indicate the
    importance of the corresponding elements in the parameter
    being pruned. If unspecified or None, the module parameter
    will be used in its place.

Returns:
      modified (i.e. pruned) version of the input module
Return type:
      module (nn.Module)
-[ Examples ]-



from torch.nn.utils import prune
CTCLoss
class torch.nn.CTCLoss(blank=0, reduction='mean', zero_infinity=False)
The Connectionist Temporal Classification loss.
Calculates loss between a continuous (unsegmented) time series and
   a target sequence. CTCLoss sums over the probability of possible
   alignments of input to target, producing a loss value which is
   differentiable with respect to each input node. The alignment of
   input to target is assumed to be "many-to-one", which limits the
   length of the target sequence such that it must be \leq the input
   length.
Parameters:
      * blank (int, optional) -- blank label. Default 0.
  * **reduction** (*str**, **optional*) -- Specifies the reduction
    to apply to the output: "'none'" | "'mean'" | "'sum'".
    "'none'": no reduction will be applied, "'mean'": the output
    losses will be divided by the target lengths and then the mean
    over the batch is taken. Default: "'mean'"
zero_infinity (bool, optional) -- Whether to zero
        infinite losses and the associated gradients. Default: "False"
        Infinite losses mainly occur when the inputs are too short to
        be aligned to the targets.

Shape:
      * Log_probs: Tensor of size (T, N, C) or (T, C), where T =
        \text{input length}, N = \text{batch size}, and C =
        \text{number of classes (including blank)}. The logarithmized
        probabilities of the outputs (e.g. obtained with
        "torch.nn.functional.log_softmax()").
  * Targets: Tensor of size (N, S) or
    (\operatorname{sum}(\text{target\_lengths})), where N =
    \text{batch size} and S = \text{max target length, if shape is
    } (N, S). It represent the target sequences. Each element in
    the target sequence is a class index. And the target index
    cannot be blank (default=0). In the (N, S) form, targets are
    padded to the length of the longest sequence, and stacked. In
the (\operatorname{sum}(\text{target_lengths})) form, the
        targets are assumed to be un-padded and concatenated within 1
        dimension.
  * Input_lengths: Tuple or tensor of size (N) or (), where N =
    \text{batch size}. It represent the lengths of the inputs
    (must each be \leq T). And the lengths are specified for each
    sequence to achieve masking under the assumption that
    sequences are padded to equal lengths.

  * Target_lengths: Tuple or tensor of size (N) or (), where N =
    \text{batch size}. It represent lengths of the targets.
    Lengths are specified for each sequence to achieve masking
    under the assumption that sequences are padded to equal
    lengths. If target shape is (N,S), target_lengths are
    effectively the stop index s_n for each target sequence, such
    that "target_n = targets[n,0:s_n]" for each target in a batch.
    Lengths must each be \leq S If the targets are given as a 1d
tensor that is the concatenation of individual targets, the
        target_lengths must add up to the total length of the tensor.
  * Output: scalar. If "reduction" is "'none'", then (N) if input
    is batched or () if input is unbatched, where N = \text{batch
    size}.

Examples:
  >>> # Target are to be padded
  >>> T = 50      # Input sequence length
  >>> C = 20      # Number of classes (including blank)
  >>> N = 16      # Batch size
  >>> S = 30      # Target sequence length of longest target in batch (padding length)
  >>> S_min = 10  # Minimum target length, for demonstration purposes
  >>>
  >>> # Initialize random batch of input vectors, for *size = (T,N,C)
  >>> input = torch.randn(T, N, C).log_softmax(2).detach().requires_grad_()
  >>>
  >>> # Initialize random batch of targets (0 = blank, 1:C = classes)
  >>> target = torch.randint(low=1, high=C, size=(N, S), dtype=torch.long)
  >>>
>>> input_lengths = torch.full(size=(N,), fill_value=T, dtype=torch.long)
  >>> target_lengths = torch.randint(low=S_min, high=S, size=(N,), dtype=torch.long)
  >>> ctc_loss = nn.CTCLoss()
  >>> loss = ctc_loss(input, target, input_lengths, target_lengths)
  >>> loss.backward()
  >>>
  >>>
  >>> # Target are to be un-padded
  >>> T = 50      # Input sequence length
  >>> C = 20      # Number of classes (including blank)
  >>> N = 16      # Batch size
  >>>
  >>> # Initialize random batch of input vectors, for *size = (T,N,C)
  >>> input = torch.randn(T, N, C).log_softmax(2).detach().requires_grad_()
  >>> input_lengths = torch.full(size=(N,), fill_value=T, dtype=torch.long)
  >>>
  >>> # Initialize random batch of targets (0 = blank, 1:C = classes)
  >>> target_lengths = torch.randint(low=1, high=T, size=(N,), dtype=torch.long)
  >>> target = torch.randint(low=1, high=C, size=(sum(target_lengths),), dtype=torch.long)
ctc_loss = nn.CTCLoss()
      >>> loss = ctc_loss(input, target, input_lengths, target_lengths)
      >>> loss.backward()
      >>>
      >>>
      >>> # Target are to be un-padded and unbatched (effectively N=1)
      >>> T = 50      # Input sequence length
      >>> C = 20      # Number of classes (including blank)
      >>>
      >>> # Initialize random batch of input vectors, for *size = (T,C)
      >>> input = torch.randn(T, C).log_softmax(2).detach().requires_grad_()
      >>> input_lengths = torch.tensor(T, dtype=torch.long)
      >>>
      >>> # Initialize random batch of targets (0 = blank, 1:C = classes)
      >>> target_lengths = torch.randint(low=1, high=T, size=(), dtype=torch.long)
      >>> target = torch.randint(low=1, high=C, size=(target_lengths,), dtype=torch.long)
      >>> ctc_loss = nn.CTCLoss()
      >>> loss = ctc_loss(input, target, input_lengths, target_lengths)
      >>> loss.backward()



Reference:
      A. Graves et al.: Connectionist Temporal Classification:
Labelling Unsegmented Sequence Data with Recurrent Neural
      Networks: https://www.cs.toronto.edu/~graves/icml_2006.pdf
Note:
 In order to use CuDNN, the following must be satisfied: "targets"
 must be in concatenated format, all "input_lengths" must be *T*.
 blank=0, "target_lengths" \leq 256, the integer arguments must be
 of dtype "torch.int32".The regular implementation uses the (more
 common in PyTorch) *torch.long* dtype.

Note:
 In some circumstances when using the CUDA backend with CuDNN,
 this operator may select a nondeterministic algorithm to increase
 performance. If this is undesirable, you can try to make the
 operation deterministic (potentially at a performance cost) by
 setting "torch.backends.cudnn.deterministic = True". Please see
 the notes on Reproducibility for background.
torch.nn.functional.unfold
torch.nn.functional.unfold(input, kernel_size, dilation=1, padding=0, stride=1)
Extracts sliding local blocks from a batched input tensor.
Warning:
 Currently, only 4-D input tensors (batched image-like tensors)
 are supported.

Warning:
 More than one element of the unfolded tensor may refer to a
 single memory location. As a result, in-place operations
 (especially ones that are vectorized) may result in incorrect
 behavior. If you need to write to the tensor, please clone it
 first.

See "torch.nn.Unfold" for details
Return type:
      Tensor
torch.masked_select
torch.masked_select(input, mask, *, out=None) -> Tensor
Returns a new 1-D tensor which indexes the "input" tensor according
   to the boolean mask "mask" which is a BoolTensor.
The shapes of the "mask" tensor and the "input" tensor don't need
   to match, but they must be broadcastable.
Note:
 The returned tensor does **not** use the same storage as the
 original tensor

Parameters:
      * input (Tensor) -- the input tensor.
  * **mask** (*BoolTensor*) -- the tensor containing the binary
    mask to index with

Keyword Arguments:
      out (Tensor, optional) -- the output tensor.
Example:
  >>> x = torch.randn(3, 4)
  >>> x
  tensor([[ 0.3552, -2.3825, -0.8297,  0.3477],
          [-1.2035,  1.2252,  0.5002,  0.6248],
          [ 0.1307, -2.0608,  0.1244,  2.0139]])
  >>> mask = x.ge(0.5)
  >>> mask
  tensor([[False, False, False, False],
          [False, True, True, True],
torch.log2
torch.log2(input, *, out=None) -> Tensor
Returns a new tensor with the logarithm to the base 2 of the
   elements of "input".
  y_{i} = \log_{2} (x_{i})

Parameters:
      input (Tensor) -- the input tensor.
Keyword Arguments:
      out (Tensor, optional) -- the output tensor.
Example:
  >>> a = torch.rand(5)
  >>> a
  tensor([ 0.8419,  0.8003,  0.9971,  0.5287,  0.0490])


  >>> torch.log2(a)
  tensor([-0.2483, -0.3213, -0.0042, -0.9196, -4.3504])
FusedMovingAvgObsFakeQuantize
class torch.quantization.fake_quantize.FusedMovingAvgObsFakeQuantize(observer=, quant_min=0, quant_max=255, **observer_kwargs)
Fused module that is used to observe the input tensor (compute
   min/max), compute scale/zero_point and fake_quantize the tensor.
   This module uses calculation similar MovingAverageMinMaxObserver
   for the inputs, to compute the min/max values in order to compute
   the scale/zero_point. The qscheme input in the observer is used to
   differentiate between symmetric/affine quantization scheme.
The output of this module is given by x_out = (clamp(round(x/scale
   + zero_point), quant_min, quant_max)-zero_point)*scale
Similar to "FakeQuantize", and accepts the same attributes as the
   base class.
quantize_qat
class torch.quantization.quantize_qat(model, run_fn, run_args, inplace=False)
Do quantization aware training and output a quantized model
Parameters:
      * model -- input model
  * **run_fn** -- a function for evaluating the prepared model,
    can be a function that simply runs the prepared model or a
    training loop

  * **run_args** -- positional arguments for *run_fn*

Returns:
      Quantized model.
Stream
class torch.cuda.Stream(device=None, priority=0, **kwargs)
Wrapper around a CUDA stream.
A CUDA stream is a linear sequence of execution that belongs to a
   specific device, independent from other streams.  See CUDA
   semantics for details.
Parameters:
      * device (torch.device or int, optional) -- a
        device on which to allocate the stream. If "device" is "None"
        (default) or a negative integer, this will use the current
        device.
  * **priority** (*int**, **optional*) -- priority of the stream.
    Can be either -1 (high priority) or 0 (low priority). By
    default, streams have priority 0.

Note:
 Although CUDA versions >= 11 support more than two levels of
 priorities, in PyTorch, we only support two levels of priorities.

query()
  Checks if all the work submitted has been completed.

  Returns:
     A boolean indicating if all kernels in this stream are
     completed.
completed.
record_event(event=None)
  Records an event.

  Parameters:
     **event** (*torch.cuda.Event**, **optional*) -- event to
     record. If not given, a new one will be allocated.

  Returns:
     Recorded event.

synchronize()
  Wait for all the kernels in this stream to complete.

  Note:

    This is a wrapper around "cudaStreamSynchronize()": see CUDA
    Stream documentation for more info.

wait_event(event)
  Makes all future work submitted to the stream wait for an event.

  Parameters:
     **event** (*torch.cuda.Event*) -- an event to wait for.

  Note:

    This is a wrapper around "cudaStreamWaitEvent()": see CUDA
    Stream documentation for more info.This function returns
    without waiting for "event": only future operations are
    affected.

wait_stream(stream)
  Synchronizes with another stream.

  All future work submitted to this stream will wait until all
torch.nn.functional.pad
torch.nn.functional.pad(input, pad, mode='constant', value=None) -> Tensor
Pads tensor.
Padding size:
      The padding size by which to pad some dimensions of "input" are
      described starting from the last dimension and moving forward.
      \left\lfloor\frac{\text{len(pad)}}{2}\right\rfloor dimensions of
      "input" will be padded. For example, to pad only the last
      dimension of the input tensor, then "pad" has the form
      (\text{padding_left}, \text{padding_right}); to pad the last 2
      dimensions of the input tensor, then use (\text{padding_left},
      \text{padding_right}, \text{padding_top},
      \text{padding_bottom}); to pad the last 3 dimensions, use
      (\text{padding_left}, \text{padding_right},
      \text{padding_top}, \text{padding_bottom}
      \text{padding_front}, \text{padding_back}).
Padding mode:
      See "torch.nn.ConstantPad2d", "torch.nn.ReflectionPad2d", and
"torch.nn.ReplicationPad2d" for concrete examples on how each of
      the padding modes works. Constant padding is implemented for
      arbitrary dimensions. Replicate and reflection padding are
      implemented for padding the last 3 dimensions of a 4D or 5D
      input tensor, the last 2 dimensions of a 3D or 4D input tensor,
      or the last dimension of a 2D or 3D input tensor.
Note:
 When using the CUDA backend, this operation may induce
 nondeterministic behaviour in its backward pass that is not
 easily switched off. Please see the notes on Reproducibility for
 background.

Parameters:
      * input (Tensor) -- N-dimensional tensor
  * **pad** (*tuple*) -- m-elements tuple, where \frac{m}{2} \leq
    input dimensions and m is even.

  * **mode** -- "'constant'", "'reflect'", "'replicate'" or
    "'circular'". Default: "'constant'"

  * **value** -- fill value for "'constant'" padding. Default: "0"

Examples:
Examples:
  >>> t4d = torch.empty(3, 3, 4, 2)
  >>> p1d = (1, 1) # pad last dim by 1 on each side
  >>> out = F.pad(t4d, p1d, "constant", 0)  # effectively zero padding
  >>> print(out.size())
  torch.Size([3, 3, 4, 4])
  >>> p2d = (1, 1, 2, 2) # pad last dim by (1, 1) and 2nd to last by (2, 2)
  >>> out = F.pad(t4d, p2d, "constant", 0)
  >>> print(out.size())
  torch.Size([3, 3, 8, 4])
  >>> t4d = torch.empty(3, 3, 4, 2)
  >>> p3d = (0, 1, 2, 1, 3, 3) # pad by (0, 1), (2, 1), and (3, 3)
  >>> out = F.pad(t4d, p3d, "constant", 0)
  >>> print(out.size())
  torch.Size([3, 9, 7, 3])
torch.Tensor.requires_grad_
Tensor.requires_grad_(requires_grad=True) -> Tensor
Change if autograd should record operations on this tensor: sets
   this tensor's "requires_grad" attribute in-place. Returns this
   tensor.
"requires_grad_()"'s main use case is to tell autograd to begin
   recording operations on a Tensor "tensor". If "tensor" has
   "requires_grad=False" (because it was obtained through a
   DataLoader, or required preprocessing or initialization),
   "tensor.requires_grad_()" makes it so that autograd will begin to
   record operations on "tensor".
Parameters:
      requires_grad (bool) -- If autograd should record
      operations on this tensor. Default: "True".
Example:
  >>> # Let's say we want to preprocess some saved weights and use
  >>> # the result as new weights.
  >>> saved_weights = [0.1, 0.2, 0.3, 0.25]
  >>> loaded_weights = torch.tensor(saved_weights)
weights = preprocess(loaded_weights)  # some function
      >>> weights
      tensor([-0.5503,  0.4926, -2.1158, -0.8303])



  >>> # Now, start to record operations done to weights
  >>> weights.requires_grad_()
  >>> out = weights.pow(2).sum()
  >>> out.backward()
  >>> weights.grad
  tensor([-1.1007,  0.9853, -4.2316, -1.6606])
torch.Tensor.contiguous
Tensor.contiguous(memory_format=torch.contiguous_format) -> Tensor
Returns a contiguous in memory tensor containing the same data as
   "self" tensor. If "self" tensor is already in the specified memory
   format, this function returns the "self" tensor.
Parameters:
      memory_format ("torch.memory_format", optional) -- the
      desired memory format of returned Tensor. Default:
      "torch.contiguous_format".
torch.std_mean
torch.std_mean(input, dim=None, *, correction=1, keepdim=False, out=None)
Calculates the standard deviation and mean over the dimensions
   specified by "dim". "dim" can be a single dimension, list of
   dimensions, or "None" to reduce over all dimensions.
The standard deviation (\sigma) is calculated as
  \sigma = \sqrt{\frac{1}{N - \delta
  N}\sum_{i=0}^{N-1}(x_i-\bar{x})^2}

where x is the sample set of elements, \bar{x} is the sample mean,
   N is the number of samples and \delta N is the "correction".
If "keepdim" is "True", the output tensor is of the same size as
   "input" except in the dimension(s) "dim" where it is of size 1.
   Otherwise, "dim" is squeezed (see "torch.squeeze()"), resulting in
   the output tensor having 1 (or "len(dim)") fewer dimension(s).
Parameters:
      * input (Tensor) -- the input tensor.
  * **dim** (*int** or **tuple of ints**, **optional*) -- the
dimension or dimensions to reduce. If "None", all dimensions
        are reduced.
Keyword Arguments:
      * correction (int) --
    difference between the sample size and sample degrees of
    freedom. Defaults to Bessel's correction, "correction=1".

    Changed in version 2.0: Previously this argument was called
    "unbiased" and was a boolean with "True" corresponding to
    "correction=1" and "False" being "correction=0".

  * **keepdim** (*bool*) -- whether the output tensor has "dim"
    retained or not.

  * **out** (*Tensor**, **optional*) -- the output tensor.

Returns:
      A tuple (std, mean) containing the standard deviation and mean.
-[ Example ]-



a = torch.tensor(
   ...     [[ 0.2035,  1.2959,  1.8101, -0.4644],
   ...      [ 1.5027, -0.3270,  0.5905,  0.6538],
   ...      [-1.5745,  1.3330, -0.5596, -0.6548],
   ...      [ 0.1264, -0.5080,  1.6420,  0.1992]])
torch.std_mean(a, dim=0, keepdim=True)
Flatten
class torch.nn.Flatten(start_dim=1, end_dim=- 1)
Flattens a contiguous range of dims into a tensor. For use with
   "Sequential".
Shape:
      * Input: (, S_{\text{start}},..., S_{i}, ..., S_{\text{end}},
        ),' where S_{i} is the size at dimension i and * means any
        number of dimensions including none.
  * Output: (*, \prod_{i=\text{start}}^{\text{end}} S_{i}, *).

Parameters:
      * start_dim (int) -- first dim to flatten (default = 1).
  * **end_dim** (*int*) -- last dim to flatten (default = -1).

Examples::
      >>> input = torch.randn(32, 1, 5, 5)
      >>> # With default parameters
      >>> m = nn.Flatten()
      >>> output = m(input)
      >>> output.size()
      torch.Size([32, 25])
      >>> # With non-default parameters
      >>> m = nn.Flatten(0, 2)
      >>> output = m(input)
      >>> output.size()
      torch.Size([160, 5])