shrinath-suresh/blogs-docs-splitted huggingface dataset:
more details.
     * **value** (*Tensor*) -- map a query and a set of key-value
       pairs to an output. See "Attention Is All You Need" for
       more details.

     * **key_padding_mask** (*Optional**[**Tensor**]*) -- if
       provided, specified padding elements in the key will be
       ignored by the attention. When given a binary mask and a
       value is True, the corresponding value on the attention
       layer will be ignored. When given a byte mask and a value
       is non-zero, the corresponding value on the attention layer
       will be ignored

     * **need_weights** (*bool*) -- output attn_output_weights.

     * **attn_mask** (*Optional**[**Tensor**]*) -- 2D or 3D mask
       that prevents attention to certain positions. A 2D mask
       will be broadcasted for all the batches while a 3D mask
       allows to specify a different mask for the entries of each
       batch.

  Return type:
batch.
  Return type:
     *Tuple*[*Tensor*, *Optional*[*Tensor*]]

  Shape:
     * Inputs:

     * query: (L, N, E) where L is the target sequence length, N
       is the batch size, E is the embedding dimension. (N, L, E)
       if "batch_first" is "True".

     * key: (S, N, E), where S is the source sequence length, N is
       the batch size, E is the embedding dimension. (N, S, E) if
       "batch_first" is "True".

     * value: (S, N, E) where S is the source sequence length, N
       is the batch size, E is the embedding dimension. (N, S, E)
       if "batch_first" is "True".

     * key_padding_mask: (N, S) where N is the batch size, S is
       the source sequence length. If a ByteTensor is provided,
       the non-zero positions will be ignored while the position
       with the zero positions will be unchanged. If a BoolTensor
       is provided, the positions with the value of "True" will be
ignored while the position with the value of "False" will
           be unchanged.
     * attn_mask: 2D mask (L, S) where L is the target sequence
       length, S is the source sequence length. 3D mask
       (N*num_heads, L, S) where N is the batch size, L is the
       target sequence length, S is the source sequence length.
       attn_mask ensure that position i is allowed to attend the
       unmasked positions. If a ByteTensor is provided, the non-
       zero positions are not allowed to attend while the zero
       positions will be unchanged. If a BoolTensor is provided,
       positions with "True" is not allowed to attend while
       "False" values will be unchanged. If a FloatTensor is
       provided, it will be added to the attention weight.

     * is_causal: If specified, applies a causal mask as attention
       mask. Mutually exclusive with providing attn_mask. Default:
       "False".
"False".
     * average_attn_weights: If true, indicates that the returned
       "attn_weights" should be averaged across heads. Otherwise,
       "attn_weights" are provided separately per head. Note that
       this flag only has an effect when "need_weights=True.".
       Default: True (i.e. average weights across heads)

     * Outputs:

     * attn_output: (L, N, E) where L is the target sequence
       length, N is the batch size, E is the embedding dimension.
       (N, L, E) if "batch_first" is "True".

     * attn_output_weights: If "average_attn_weights=True",
       returns attention weights averaged across heads of shape
       (N, L, S), where N is the batch size, L is the target
       sequence length, S is the source sequence length. If
       "average_attn_weights=False", returns attention weights per
       head of shape (N, num_heads, L, S).
torch.ge
torch.ge(input, other, *, out=None) -> Tensor
Computes \text{input} \geq \text{other} element-wise.
The second argument can be a number or a tensor whose shape is
   broadcastable with the first argument.
Parameters:
      * input (Tensor) -- the tensor to compare
  * **other** (*Tensor** or **float*) -- the tensor or value to
    compare

Keyword Arguments:
      out (Tensor, optional) -- the output tensor.
Returns:
      A boolean tensor that is True where "input" is greater than or
      equal to "other" and False elsewhere
Example:
  >>> torch.ge(torch.tensor([[1, 2], [3, 4]]), torch.tensor([[1, 1], [4, 4]]))
  tensor([[True, True], [False, True]])
Conv2d
class torch.ao.nn.qat.Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True, padding_mode='zeros', qconfig=None, device=None, dtype=None)
A Conv2d module attached with FakeQuantize modules for weight, used
   for quantization aware training.
We adopt the same interface as torch.nn.Conv2d, please see https
   ://pytorch.org/docs/stable/nn.html?highlight=conv2d#torch.nn.Conv2d
   for documentation.
Similar to torch.nn.Conv2d, with FakeQuantize modules initialized
   to default.
Variables:
      weight_fake_quant -- fake quant module for weight
linear
class torch.ao.nn.quantized.functional.linear(input, weight, bias=None, scale=None, zero_point=None)
Applies a linear transformation to the incoming quantized data: y =
   xA^T + b. See "Linear"
Note:
 Current implementation packs weights on every call, which has
 penalty on performance. If you want to avoid the overhead, use
 "Linear".

Parameters:
      * input (Tensor) -- Quantized input of type torch.quint8
  * **weight** (*Tensor*) -- Quantized weight of type
    *torch.qint8*

  * **bias** (*Tensor*) -- None or fp32 bias of type *torch.float*

  * **scale** (*double*) -- output scale. If None, derived from
    the input scale

  * **zero_point** (*python:long*) -- output zero point. If None,
    derived from the input zero_point

Return type:
      Tensor
Shape:
      * Input: (N, *, in_features) where *** means any number of
        additional dimensions
  * Weight: (out\_features, in\_features)
MovingAveragePerChannelMinMaxObserver
class torch.quantization.observer.MovingAveragePerChannelMinMaxObserver(averaging_constant=0.01, ch_axis=0, dtype=torch.quint8, qscheme=torch.per_channel_affine, reduce_range=False, quant_min=None, quant_max=None, eps=1.1920928955078125e-07, **kwargs)
Observer module for computing the quantization parameters based on
   the running per channel min and max values.
This observer uses the tensor min/max statistics to compute the per
   channel quantization parameters. The module records the running
   minimum and maximum of incoming tensors, and uses this statistic to
   compute the quantization parameters.
Parameters:
      * averaging_constant -- Averaging constant for min/max.
  * **ch_axis** -- Channel axis

  * **dtype** -- Quantized data type

  * **qscheme** -- Quantization scheme to be used

  * **reduce_range** -- Reduces the range of the quantized data
    type by 1 bit
type by 1 bit
  * **quant_min** -- Minimum quantization value. If unspecified,
    it will follow the 8-bit setup.

  * **quant_max** -- Maximum quantization value. If unspecified,
    it will follow the 8-bit setup.

  * **eps** (*Tensor*) -- Epsilon value for float32, Defaults to
    *torch.finfo(torch.float32).eps*.

The quantization parameters are computed the same way as in
   "MovingAverageMinMaxObserver", with the difference that the running
   min/max values are stored per channel. Scales and zero points are
   thus computed per channel as well.
Note:
 If the running minimum equals to the running maximum, the scales
 and zero_points are set to 1.0 and 0.
torch.nn.functional.hardswish
torch.nn.functional.hardswish(input, inplace=False)
Applies the hardswish function, element-wise, as described in the
   paper:
Searching for MobileNetV3.
  \text{Hardswish}(x) = \begin{cases}     0 & \text{if~} x \le -3,
  \\     x & \text{if~} x \ge +3, \\     x \cdot (x + 3) /6 &
  \text{otherwise} \end{cases}

See "Hardswish" for more details.
Return type:
      Tensor
torch.quantized_batch_norm
torch.quantized_batch_norm(input, weight=None, bias=None, mean, var, eps, output_scale, output_zero_point) -> Tensor
Applies batch normalization on a 4D (NCHW) quantized tensor.
  y = \frac{x - \mathrm{E}[x]}{\sqrt{\mathrm{Var}[x] + \epsilon}}
  * \gamma + \beta

Parameters:
      * input (Tensor) -- quantized tensor
  * **weight** (*Tensor*) -- float tensor that corresponds to the
    gamma, size C

  * **bias** (*Tensor*) -- float tensor that corresponds to the
    beta, size C

  * **mean** (*Tensor*) -- float mean value in batch
    normalization, size C

  * **var** (*Tensor*) -- float tensor for variance, size C

  * **eps** (*float*) -- a value added to the denominator for
    numerical stability.

  * **output_scale** (*float*) -- output quantized tensor scale

  * **output_zero_point** (*int*) -- output quantized tensor
    zero_point

Returns:
zero_point
Returns:
      A quantized tensor with batch normalization applied.
Return type:
      Tensor
Example:
  >>> qx = torch.quantize_per_tensor(torch.rand(2, 2, 2, 2), 1.5, 3, torch.quint8)
  >>> torch.quantized_batch_norm(qx, torch.ones(2), torch.zeros(2), torch.rand(2), torch.rand(2), 0.00001, 0.2, 2)
  tensor([[[[-0.2000, -0.2000],
        [ 1.6000, -0.2000]],

       [[-0.4000, -0.4000],
        [-0.4000,  0.6000]]],


      [[[-0.2000, -0.2000],
        [-0.2000, -0.2000]],

       [[ 0.6000, -0.4000],
        [ 0.6000, -0.4000]]]], size=(2, 2, 2, 2), dtype=torch.quint8,
     quantization_scheme=torch.per_tensor_affine, scale=0.2, zero_point=2)
torch.linalg.cholesky_ex
torch.linalg.cholesky_ex(A, *, upper=False, check_errors=False, out=None)
Computes the Cholesky decomposition of a complex Hermitian or real
   symmetric positive-definite matrix.
This function skips the (slow) error checking and error message
   construction of "torch.linalg.cholesky()", instead directly
   returning the LAPACK error codes as part of a named tuple "(L,
   info)". This makes this function a faster way to check if a matrix
   is positive-definite, and it provides an opportunity to handle
   decomposition errors more gracefully or performantly than
   "torch.linalg.cholesky()" does.
Supports input of float, double, cfloat and cdouble dtypes. Also
   supports batches of matrices, and if "A" is a batch of matrices
   then the output has the same batch dimensions.
If "A" is not a Hermitian positive-definite matrix, or if it's a
   batch of matrices and one or more of them is not a Hermitian
positive-definite matrix, then "info" stores a positive integer for
   the corresponding matrix. The positive integer indicates the order
   of the leading minor that is not positive-definite, and the
   decomposition could not be completed. "info" filled with zeros
   indicates that the decomposition was successful. If
   "check_errors=True" and "info" contains positive integers, then a
   RuntimeError is thrown.
Note:
 When the inputs are on a CUDA device, this function synchronizes
 only when "check_errors"*= True*.

Warning:
 This function is "experimental" and it may change in a future
 PyTorch release.

See also:
 "torch.linalg.cholesky()" is a NumPy compatible variant that
 always checks for errors.

Parameters:
      A (Tensor) -- the Hermitian n times n matrix or the
      batch of such matrices of size (, n, n)* where *** is one or
      more batch dimensions.
Keyword Arguments:
Keyword Arguments:
      * upper (bool, optional) -- whether to return an upper
        triangular matrix. The tensor returned with upper=True is the
        conjugate transpose of the tensor returned with upper=False.
  * **check_errors** (*bool**, **optional*) -- controls whether to
    check the content of "infos". Default: *False*.

  * **out** (*tuple**, **optional*) -- tuple of two tensors to
    write the output to. Ignored if *None*. Default: *None*.

Examples:
  >>> A = torch.randn(2, 2, dtype=torch.complex128)
  >>> A = A @ A.t().conj()  # creates a Hermitian positive-definite matrix
  >>> L, info = torch.linalg.cholesky_ex(A)
  >>> A
  tensor([[ 2.3792+0.0000j, -0.9023+0.9831j],
          [-0.9023-0.9831j,  0.8757+0.0000j]], dtype=torch.complex128)
  >>> L
  tensor([[ 1.5425+0.0000j,  0.0000+0.0000j],
          [-0.5850-0.6374j,  0.3567+0.0000j]], dtype=torch.complex128)
  >>> info
  tensor(0, dtype=torch.int32)
torch.Tensor.sparse_mask
Tensor.sparse_mask(mask) -> Tensor
Returns a new sparse tensor with values from a strided tensor
   "self" filtered by the indices of the sparse tensor "mask". The
   values of "mask" sparse tensor are ignored. "self" and "mask"
   tensors must have the same shape.
Note:
 The returned sparse tensor might contain duplicate values if
 "mask" is not coalesced. It is therefore advisable to pass
 "mask.coalesce()" if such behavior is not desired.

Note:
 The returned sparse tensor has the same indices as the sparse
 tensor "mask", even when the corresponding values in "self" are
 zeros.

Parameters:
      mask (Tensor) -- a sparse tensor whose indices are used as
      a filter
Example:
  >>> nse = 5
  >>> dims = (5, 5, 2, 2)
  >>> I = torch.cat([torch.randint(0, dims[0], size=(nse,)),
  ...                torch.randint(0, dims[1], size=(nse,))], 0).reshape(2, nse)
V = torch.randn(nse, dims[2], dims[3])
      >>> S = torch.sparse_coo_tensor(I, V, dims).coalesce()
      >>> D = torch.randn(dims)
      >>> D.sparse_mask(S)
      tensor(indices=tensor([[0, 0, 0, 2],
                             [0, 1, 4, 3]]),
             values=tensor([[[ 1.6550,  0.2397],
                             [-0.1611, -0.0779]],



                        [[ 0.2326, -1.0558],
                         [ 1.4711,  1.9678]],

                        [[-0.5138, -0.0411],
                         [ 1.9417,  0.5158]],

                        [[ 0.0793,  0.0036],
                         [-0.2569, -0.1055]]]),
         size=(5, 5, 2, 2), nnz=4, layout=torch.sparse_coo)
KLDivLoss
class torch.nn.KLDivLoss(size_average=None, reduce=None, reduction='mean', log_target=False)
The Kullback-Leibler divergence loss.
For tensors of the same shape y_{\text{pred}},\ y_{\text{true}},
   where y_{\text{pred}} is the "input" and y_{\text{true}} is the
   "target", we define the pointwise KL-divergence as
  L(y_{\text{pred}},\ y_{\text{true}})     = y_{\text{true}} \cdot
  \log \frac{y_{\text{true}}}{y_{\text{pred}}}     =
  y_{\text{true}} \cdot (\log y_{\text{true}} - \log
  y_{\text{pred}})

To avoid underflow issues when computing this quantity, this loss
   expects the argument "input" in the log-space. The argument
   "target" may also be provided in the log-space if "log_target"=
   True.
To summarise, this function is roughly equivalent to computing
  if not log_target: # default
      loss_pointwise = target * (target.log() - input)
  else:
      loss_pointwise = target.exp() * (target - input)
and then reducing this result depending on the argument "reduction"
   as
  if reduction == "mean":  # default
      loss = loss_pointwise.mean()
  elif reduction == "batchmean":  # mathematically correct
      loss = loss_pointwise.sum() / input.size(0)
  elif reduction == "sum":
      loss = loss_pointwise.sum()
  else:  # reduction == "none"
      loss = loss_pointwise

Note:
 As all the other losses in PyTorch, this function expects the
 first argument, "input", to be the output of the model (e.g. the
 neural network) and the second, "target", to be the observations
 in the dataset. This differs from the standard mathematical
 notation KL(P\ ||\ Q) where P denotes the distribution of the
 observations and Q denotes the model.

Warning:
 "reduction"*= "mean"* doesn't return the true KL divergence
 value, please use "reduction"*= "batchmean"* which aligns with
 the mathematical definition. In a future release, *"mean"* will
be changed to be the same as "batchmean".
Parameters:
      * size_average (bool, optional) -- Deprecated (see
        "reduction"). By default, the losses are averaged over each
        loss element in the batch. Note that for some losses, there
        are multiple elements per sample. If the field "size_average"
        is set to False, the losses are instead summed for each
        minibatch. Ignored when "reduce" is False. Default: True
  * **reduce** (*bool**, **optional*) -- Deprecated (see
    "reduction"). By default, the losses are averaged or summed
    over observations for each minibatch depending on
    "size_average". When "reduce" is *False*, returns a loss per
    batch element instead and ignores "size_average". Default:
    *True*

  * **reduction** (*str**, **optional*) -- Specifies the reduction
    to apply to the output. Default: *"mean"*

  * **log_target** (*bool**, **optional*) -- Specifies whether
target is the log space. Default: False
Shape:
      * Input: (*), where * means any number of dimensions.
  * Target: (*), same shape as the input.

  * Output: scalar by default. If "reduction" is *'none'*, then
    (*), same shape as the input.

Examples:
  >>> import torch.nn.functional as F
  >>> kl_loss = nn.KLDivLoss(reduction="batchmean")
  >>> # input should be a distribution in the log space
  >>> input = F.log_softmax(torch.randn(3, 5, requires_grad=True), dim=1)
  >>> # Sample a batch of distributions. Usually this would come from the dataset
  >>> target = F.softmax(torch.rand(3, 5), dim=1)
  >>> output = kl_loss(input, target)

  >>> kl_loss = nn.KLDivLoss(reduction="batchmean", log_target=True)
  >>> log_target = F.log_softmax(torch.rand(3, 5), dim=1)
  >>> output = kl_loss(input, log_target)
ChainedScheduler
class torch.optim.lr_scheduler.ChainedScheduler(schedulers)
Chains list of learning rate schedulers. It takes a list of
   chainable learning rate schedulers and performs consecutive step()
   functions belonging to them by just one call.
Parameters:
      schedulers (list) -- List of chained schedulers.
-[ Example ]-



Assuming optimizer uses lr = 1. for all groups
lr = 0.09     if epoch == 0
lr = 0.081    if epoch == 1
lr = 0.729    if epoch == 2
lr = 0.6561   if epoch == 3
lr = 0.59049  if epoch >= 4
scheduler1 = ConstantLR(self.opt, factor=0.1, total_iters=2)
scheduler2 = ExponentialLR(self.opt, gamma=0.9)
scheduler = ChainedScheduler([scheduler1, scheduler2])
for epoch in range(100):
    train(...)
    validate(...)
    scheduler.step()



get_last_lr()
  Return last computed learning rate by current scheduler.

load_state_dict(state_dict)
load_state_dict(state_dict)
  Loads the schedulers state.

  Parameters:
     **state_dict** (*dict*) -- scheduler state. Should be an
     object returned from a call to "state_dict()".

print_lr(is_verbose, group, lr, epoch=None)
  Display the current learning rate.

state_dict()
  Returns the state of the scheduler as a "dict".

  It contains an entry for every variable in self.__dict__ which
  is not the optimizer. The wrapped scheduler states will also be
  saved.
torch.fmax
torch.fmax(input, other, *, out=None) -> Tensor
Computes the element-wise maximum of "input" and "other".
This is like "torch.maximum()" except it handles NaNs differently:
   if exactly one of the two elements being compared is a NaN then the
   non-NaN element is taken as the maximum. Only if both elements are
   NaN is NaN propagated.
This function is a wrapper around C++'s "std::fmax" and is similar
   to NumPy's "fmax" function.
Supports broadcasting to a common shape, type promotion, and
   integer and floating-point inputs.
Parameters:
      * input (Tensor) -- the input tensor.
  * **other** (*Tensor*) -- the second input tensor

Keyword Arguments:
      out (Tensor, optional) -- the output tensor.
Example:
  >>> a = torch.tensor([9.7, float('nan'), 3.1, float('nan')])
  >>> b = torch.tensor([-2.2, 0.5, float('nan'), float('nan')])
  >>> torch.fmax(a, b)
  tensor([9.7000, 0.5000, 3.1000,    nan])
torch.acosh
torch.acosh(input, *, out=None) -> Tensor
Returns a new tensor with the inverse hyperbolic cosine of the
   elements of "input".
  \text{out}_{i} = \cosh^{-1}(\text{input}_{i})

Note:
 The domain of the inverse hyperbolic cosine is *[1, inf)* and
 values outside this range will be mapped to "NaN", except for *+
 INF* for which the output is mapped to *+ INF*.

Parameters:
      input (Tensor) -- the input tensor.
Keyword Arguments:
      out (Tensor, optional) -- the output tensor.
Example:
  >>> a = torch.randn(4).uniform_(1, 2)
  >>> a
  tensor([ 1.3192, 1.9915, 1.9674, 1.7151 ])
  >>> torch.acosh(a)
  tensor([ 0.7791, 1.3120, 1.2979, 1.1341 ])
torch.div
torch.div(input, other, *, rounding_mode=None, out=None) -> Tensor
Divides each element of the input "input" by the corresponding
   element of "other".
  \text{out}_i = \frac{\text{input}_i}{\text{other}_i}

Note:
 By default, this performs a "true" division like Python 3. See
 the "rounding_mode" argument for floor division.

Supports broadcasting to a common shape, type promotion, and
   integer, float, and complex inputs. Always promotes integer types
   to the default scalar type.
Parameters:
      * input (Tensor) -- the dividend
  * **other** (*Tensor** or **Number*) -- the divisor

Keyword Arguments:
      * rounding_mode (str, optional) --
    Type of rounding applied to the result:

    * None - default behavior. Performs no rounding and, if both
      "input" and "other" are integer types, promotes the inputs
      to the default scalar type. Equivalent to true division in
Python (the "/" operator) and NumPy's "np.true_divide".
    * ""trunc"" - rounds the results of the division towards zero.
      Equivalent to C-style integer division.

    * ""floor"" - rounds the results of the division down.
      Equivalent to floor division in Python (the "//" operator)
      and NumPy's "np.floor_divide".

  * **out** (*Tensor**, **optional*) -- the output tensor.

Examples:
  >>> x = torch.tensor([ 0.3810,  1.2774, -0.2972, -0.3719,  0.4637])
  >>> torch.div(x, 0.5)
  tensor([ 0.7620,  2.5548, -0.5944, -0.7438,  0.9274])

  >>> a = torch.tensor([[-0.3711, -1.9353, -0.4605, -0.2917],
  ...                   [ 0.1815, -1.0111,  0.9805, -1.5923],
  ...                   [ 0.1062,  1.4581,  0.7759, -1.2344],
  ...                   [-0.1830, -0.0313,  1.1908, -1.4757]])
  >>> b = torch.tensor([ 0.8032,  0.2930, -0.8113, -0.2308])
  >>> torch.div(a, b)
  tensor([[-0.4620, -6.6051,  0.5676,  1.2639],
[ 0.2260, -3.4509, -1.2086,  6.8990],
              [ 0.1322,  4.9764, -0.9564,  5.3484],
              [-0.2278, -0.1068, -1.4678,  6.3938]])
  >>> torch.div(a, b, rounding_mode='trunc')
  tensor([[-0., -6.,  0.,  1.],
          [ 0., -3., -1.,  6.],
          [ 0.,  4., -0.,  5.],
          [-0., -0., -1.,  6.]])

  >>> torch.div(a, b, rounding_mode='floor')
  tensor([[-1., -7.,  0.,  1.],
          [ 0., -4., -2.,  6.],
          [ 0.,  4., -1.,  5.],
          [-1., -1., -2.,  6.]])
torch.Tensor.index_reduce_
Tensor.index_reduce_(dim, index, source, reduce, *, include_self=True) -> Tensor
Accumulate the elements of "source" into the "self" tensor by
   accumulating to the indices in the order given in "index" using the
   reduction given by the "reduce" argument. For example, if "dim ==
   0", "index[i] == j", "reduce == prod" and "include_self == True"
   then the "i"th row of "source" is multiplied by the "j"th row of
   "self". If "include_self="True"", the values in the "self" tensor
   are included in the reduction, otherwise, rows in the "self" tensor
   that are accumulated to are treated as if they were filled with the
   reduction identites.
The "dim"th dimension of "source" must have the same size as the
   length of "index" (which must be a vector), and all other
   dimensions must match "self", or an error will be raised.
For a 3-D tensor with "reduce="prod"" and "include_self=True" the
   output is given as: