shrinath-suresh/blogs-docs-splitted huggingface dataset:
as the minimum size big enough to hold all non-zero blocks.
Keyword Arguments:
      * dtype ("torch.dtype", optional) -- the desired data type
        of returned tensor.  Default: if None, infers data type from
        "values".
  * **device** ("torch.device", optional) -- the desired device of
    returned tensor.  Default: if None, uses the current device
    for the default tensor type (see
    "torch.set_default_tensor_type()"). "device" will be the CPU
    for CPU tensor types and the current CUDA device for CUDA
    tensor types.

  * **requires_grad** (*bool**, **optional*) -- If autograd should
    record operations on the returned tensor. Default: "False".

  * **check_invariants** (*bool**, **optional*) -- If sparse
    tensor invariants are checked. Default: as returned by
    "torch.sparse.check_sparse_tensor_invariants.is_enabled()",
    initially False.

Example::
      >>> crow_indices = [0, 1, 2]
Example::
      >>> crow_indices = [0, 1, 2]
      >>> col_indices = [0, 1]
      >>> values = [[[1, 2], [3, 4]], [[5, 6], [7, 8]]]
      >>> torch.sparse_bsr_tensor(torch.tensor(crow_indices, dtype=torch.int64),
      ...                         torch.tensor(col_indices, dtype=torch.int64),
      ...                         torch.tensor(values), dtype=torch.double)
      tensor(crow_indices=tensor([0, 1, 2]),
             col_indices=tensor([0, 1]),
             values=tensor([[[1., 2.],
                             [3., 4.]],
                            [[5., 6.],
                             [7., 8.]]]), size=(2, 2), nnz=2, dtype=torch.float64,
             layout=torch.sparse_bsr)
LogSoftmax
class torch.nn.LogSoftmax(dim=None)
Applies the \log(\text{Softmax}(x)) function to an n-dimensional
   input Tensor. The LogSoftmax formulation can be simplified as:
  \text{LogSoftmax}(x_{i}) = \log\left(\frac{\exp(x_i) }{ \sum_j
  \exp(x_j)} \right)

Shape:
      * Input: (*) where *** means, any number of additional
        dimensions
  * Output: (*), same shape as the input

Parameters:
      dim (int) -- A dimension along which LogSoftmax will be
      computed.
Returns:
      a Tensor of the same dimension and shape as the input with
      values in the range [-inf, 0)
Return type:
      None
Examples:
  >>> m = nn.LogSoftmax(dim=1)
  >>> input = torch.randn(2, 3)
  >>> output = m(input)
torch.func.jvp
torch.func.jvp(func, primals, tangents, *, strict=False, has_aux=False)
Standing for the Jacobian-vector product, returns a tuple
   containing the output of func(primals)* and the "Jacobian of
   "func" evaluated at "primals"" times "tangents". This is also known
   as forward-mode autodiff.
Parameters:
      * func (function) -- A Python function that takes one or
        more arguments, one of which must be a Tensor, and returns one
        or more Tensors
  * **primals** (*Tensors*) -- Positional arguments to "func" that
    must all be Tensors. The returned function will also be
    computing the derivative with respect to these arguments

  * **tangents** (*Tensors*) -- The "vector" for which Jacobian-
    vector-product is computed. Must be the same structure and
    sizes as the inputs to "func".

  * **has_aux** (*bool*) -- Flag indicating that "func" returns a
"(output, aux)" tuple where the first element is the output of
        the function to be differentiated and the second element is
        other auxiliary objects that will not be differentiated.
        Default: False.
Returns:
      Returns a "(output, jvp_out)" tuple containing the output of
      "func" evaluated at "primals" and the Jacobian-vector product.
      If "has_aux is True", then instead returns a "(output, jvp_out,
      aux)" tuple.
Note:
 You may see this API error out with "forward-mode AD not
 implemented for operator X". If so, please file a bug report and
 we will prioritize it.

jvp is useful when you wish to compute gradients of a function R^1
   -> R^N



from torch.func import jvp
x = torch.randn([])
f = lambda x: x * torch.tensor([1., 2., 3])
value, grad = jvp(f, (x,), (torch.tensor(1.),))
assert torch.allclose(value, f(x))
assert torch.allclose(grad, torch.tensor([1., 2, 3]))
"jvp()" can support functions with multiple inputs by passing in
   the tangents for each of the inputs



from torch.func import jvp
x = torch.randn(5)
y = torch.randn(5)
f = lambda x, y: (x * y)
_, output = jvp(f, (x, y), (torch.ones(5), torch.ones(5)))
assert torch.allclose(output, x + y)
torch.angle
torch.angle(input, *, out=None) -> Tensor
Computes the element-wise angle (in radians) of the given "input"
   tensor.
  \text{out}_{i} = angle(\text{input}_{i})

Parameters:
      input (Tensor) -- the input tensor.
Keyword Arguments:
      out (Tensor, optional) -- the output tensor.
Note:
 Starting in PyTorch 1.8, angle returns pi for negative real
 numbers, zero for non-negative real numbers, and propagates NaNs.
 Previously the function would return zero for all real numbers
 and not propagate floating-point NaNs.

Example:
  >>> torch.angle(torch.tensor([-1 + 1j, -2 + 2j, 3 - 3j]))*180/3.14159
  tensor([ 135.,  135,  -45])
torch.flipud
torch.flipud(input) -> Tensor
Flip tensor in the up/down direction, returning a new tensor.
Flip the entries in each column in the up/down direction. Rows are
   preserved, but appear in a different order than before.
Note:
 Requires the tensor to be at least 1-D.

Note:
 *torch.flipud* makes a copy of "input"'s data. This is different
 from NumPy's *np.flipud*, which returns a view in constant time.
 Since copying a tensor's data is more work than viewing that
 data, *torch.flipud* is expected to be slower than *np.flipud*.

Parameters:
      input (Tensor) -- Must be at least 1-dimensional.
Example:
  >>> x = torch.arange(4).view(2, 2)
  >>> x
  tensor([[0, 1],
          [2, 3]])
  >>> torch.flipud(x)
  tensor([[2, 3],
          [0, 1]])
torch.cartesian_prod
torch.cartesian_prod(*tensors)
Do cartesian product of the given sequence of tensors. The behavior
   is similar to python's itertools.product.
Parameters:
      tensors (Tensor*) -- any number of 1 dimensional tensors.
Returns:
      A tensor equivalent to converting all the input tensors into
      lists, do itertools.product on these lists, and finally
      convert the resulting list into tensor.
Return type:
      Tensor
Example:
  >>> import itertools
  >>> a = [1, 2, 3]
  >>> b = [4, 5]
  >>> list(itertools.product(a, b))
  [(1, 4), (1, 5), (2, 4), (2, 5), (3, 4), (3, 5)]
  >>> tensor_a = torch.tensor(a)
  >>> tensor_b = torch.tensor(b)
  >>> torch.cartesian_prod(tensor_a, tensor_b)
  tensor([[1, 4],
          [1, 5],
          [2, 4],
          [2, 5],
          [3, 4],
          [3, 5]])
torch.autograd.profiler.profile.key_averages
profile.key_averages(group_by_input_shape=False, group_by_stack_n=0)
Averages all function events over their keys.
Parameters:
      * group_by_input_shapes -- group entries by (event name,
        input shapes) rather than just event name. This is useful to
        see which input shapes contribute to the runtime the most and
        may help with size-specific optimizations or choosing the best
        candidates for quantization (aka fitting a roof line)
  * **group_by_stack_n** -- group by top n stack trace entries

Returns:
      An EventList containing FunctionEventAvg objects.
Hardsigmoid
class torch.nn.Hardsigmoid(inplace=False)
Applies the Hardsigmoid function element-wise.
Hardsigmoid is defined as:
  \text{Hardsigmoid}(x) = \begin{cases}     0 & \text{if~} x \le
  -3, \\     1 & \text{if~} x \ge +3, \\     x / 6 + 1 / 2 &
  \text{otherwise} \end{cases}

Parameters:
      inplace (bool) -- can optionally do the operation in-
      place. Default: "False"
Shape:
      * Input: (*), where * means any number of dimensions.
  * Output: (*), same shape as the input.

[image]
Examples:
  >>> m = nn.Hardsigmoid()
  >>> input = torch.randn(2)
  >>> output = m(input)
torch.linalg.vander
torch.linalg.vander(x, N=None) -> Tensor
Generates a Vandermonde matrix.
Returns the Vandermonde matrix V
  V = \begin{pmatrix}         1 & x_1 & x_1^2 & \dots &
  x_1^{N-1}\\         1 & x_2 & x_2^2 & \dots & x_2^{N-1}\\
  1 & x_3 & x_3^2 & \dots & x_3^{N-1}\\         \vdots & \vdots &
  \vdots & \ddots &\vdots \\         1 & x_n & x_n^2 & \dots &
  x_n^{N-1}     \end{pmatrix}.

for N > 1. If "N"= None, then N = x.size(-1) so that the
   output is a square matrix.
Supports inputs of float, double, cfloat, cdouble, and integral
   dtypes. Also supports batches of vectors, and if "x" is a batch of
   vectors then the output has the same batch dimensions.
Differences with numpy.vander:

Unlike numpy.vander, this function returns the powers of "x" in
     ascending order. To get them in the reverse order call
     "linalg.vander(x, N).flip(-1)".

Parameters:
Parameters:
      x (Tensor) -- tensor of shape (, n)* where *** is zero
      or more batch dimensions consisting of vectors.
Keyword Arguments:
      N (int, optional) -- Number of columns in the output.
      Default: x.size(-1)
Example:
  >>> x = torch.tensor([1, 2, 3, 5])
  >>> linalg.vander(x)
  tensor([[  1,   1,   1,   1],
          [  1,   2,   4,   8],
          [  1,   3,   9,  27],
          [  1,   5,  25, 125]])
  >>> linalg.vander(x, N=3)
  tensor([[ 1,  1,  1],
          [ 1,  2,  4],
          [ 1,  3,  9],
          [ 1,  5, 25]])
torch.nn.functional.silu
torch.nn.functional.silu(input, inplace=False)
Applies the Sigmoid Linear Unit (SiLU) function, element-wise. The
   SiLU function is also known as the swish function.
  \text{silu}(x) = x * \sigma(x), \text{where } \sigma(x) \text{
  is the logistic sigmoid.}

Note:
 See Gaussian Error Linear Units (GELUs) where the SiLU (Sigmoid
 Linear Unit) was originally coined, and see Sigmoid-Weighted
 Linear Units for Neural Network Function Approximation in
 Reinforcement Learning and Swish: a Self-Gated Activation
 Function where the SiLU was experimented with later.

See "SiLU" for more details.
Return type:
      Tensor
torch.clone
torch.clone(input, *, memory_format=torch.preserve_format) -> Tensor
Returns a copy of "input".
Note:
 This function is differentiable, so gradients will flow back from
 the result of this operation to "input". To create a tensor
 without an autograd relationship to "input" see "detach()".

Parameters:
      input (Tensor) -- the input tensor.
Keyword Arguments:
      memory_format ("torch.memory_format", optional) -- the
      desired memory format of returned tensor. Default:
      "torch.preserve_format".
LinearReLU
class torch.ao.nn.intrinsic.qat.LinearReLU(in_features, out_features, bias=True, qconfig=None)
A LinearReLU module fused from Linear and ReLU modules, attached
   with FakeQuantize modules for weight, used in quantization aware
   training.
We adopt the same interface as "torch.nn.Linear".
Similar to torch.nn.intrinsic.LinearReLU, with FakeQuantize
   modules initialized to default.
Variables:
      weight (torch.Tensor) -- fake quant module for weight
Examples:
  >>> m = nn.qat.LinearReLU(20, 30)
  >>> input = torch.randn(128, 20)
  >>> output = m(input)
  >>> print(output.size())
  torch.Size([128, 30])
torch.imag
torch.imag(input) -> Tensor
Returns a new tensor containing imaginary values of the "self"
   tensor. The returned tensor and "self" share the same underlying
   storage.
Warning:
 "imag()" is only supported for tensors with complex dtypes.

Parameters:
      input (Tensor) -- the input tensor.
Example:
  >>> x=torch.randn(4, dtype=torch.cfloat)
  >>> x
  tensor([(0.3100+0.3553j), (-0.5445-0.7896j), (-1.6492-0.0633j), (-0.0638-0.8119j)])
  >>> x.imag
  tensor([ 0.3553, -0.7896, -0.0633, -0.8119])
RMSprop
class torch.optim.RMSprop(params, lr=0.01, alpha=0.99, eps=1e-08, weight_decay=0, momentum=0, centered=False, foreach=None, maximize=False, differentiable=False)
Implements RMSprop algorithm.
  \begin{aligned}      &\rule{110mm}{0.4pt}
  \\      &\textbf{input}      : \alpha \text{ (alpha)},\: \gamma
  \text{ (lr)},          \: \theta_0 \text{ (params)}, \:
  f(\theta) \text{ (objective)}                   \\
  &\hspace{13mm}   \lambda \text{ (weight decay)},\: \mu \text{
  (momentum)},\: centered\\      &\textbf{initialize} : v_0
  \leftarrow 0 \text{ (square average)}, \:          \textbf{b}_0
  \leftarrow 0 \text{ (buffer)}, \: g^{ave}_0 \leftarrow 0
  \\[-1.ex]      &\rule{110mm}{0.4pt}
  \\      &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \:
  \textbf{do}                         \\      &\hspace{5mm}g_t
  \leftarrow   \nabla_{\theta} f_t (\theta_{t-1})           \\
  &\hspace{5mm}if \: \lambda \neq 0
&\hspace{5mm}if \: \lambda \neq 0
      \      &\hspace{10mm} g_t \leftarrow g_t + \lambda
      \theta_{t-1}                            \      &\hspace{5mm}v_t
      \leftarrow   \alpha v_{t-1} + (1 - \alpha) g^2_t
      \hspace{8mm}
      \      &\hspace{5mm} \tilde{v_t} \leftarrow v_t
      \      &\hspace{5mm}if \: centered
      \      &\hspace{10mm} g^{ave}t \leftarrow g^{ave} \alpha
      + (1-\alpha) g_t            \      &\hspace{10mm} \tilde{v_t}
      \leftarrow \tilde{v_t} -  \big(g^{ave}{t} \big)^2        \
      &\hspace{5mm}if \: \mu > 0
      \      &\hspace{10mm} \textbf{b}_t\leftarrow \mu
      \textbf{b} +          g_t/ \big(\sqrt{\tilde{v_t}} +
      \epsilon \big)                                   \
      &\hspace{10mm} \theta_t \leftarrow \theta_{t-1} - \gamma
      \textbf{b}t                \      &\hspace{5mm} else
      \      &\hspace{10mm}\theta_t      \leftarrow   \theta -
      \gamma  g_t/ \big(\sqrt{\tilde{v_t}} + \epsilon \big)
\hspace{3mm}              \      &\rule{110mm}{0.4pt}
      \[-1.ex]      &\bf{return} \:  \theta_t
      \[-1.ex]      &\rule{110mm}{0.4pt}
      \[-1.ex] \end{aligned}
For further details regarding the algorithm we refer to lecture
   notes by G. Hinton. and centered version Generating Sequences With
   Recurrent Neural Networks. The implementation here takes the square
   root of the gradient average before adding epsilon (note that
   TensorFlow interchanges these two operations). The effective
   learning rate is thus \gamma/(\sqrt{v} + \epsilon) where \gamma is
   the scheduled learning rate and v is the weighted moving average of
   the squared gradient.
Parameters:
      * params (iterable) -- iterable of parameters to optimize
        or dicts defining parameter groups
  * **lr** (*float**, **optional*) -- learning rate (default:
    1e-2)

  * **momentum** (*float**, **optional*) -- momentum factor
    (default: 0)
(default: 0)
  * **alpha** (*float**, **optional*) -- smoothing constant
    (default: 0.99)

  * **eps** (*float**, **optional*) -- term added to the
    denominator to improve numerical stability (default: 1e-8)

  * **centered** (*bool**, **optional*) -- if "True", compute the
    centered RMSProp, the gradient is normalized by an estimation
    of its variance

  * **weight_decay** (*float**, **optional*) -- weight decay (L2
    penalty) (default: 0)

  * **foreach** (*bool**, **optional*) -- whether foreach
    implementation of optimizer is used. If unspecified by the
    user (so foreach is None), we will try to use foreach over the
    for-loop implementation on CUDA, since it is usually
    significantly more performant. (default: None)

  * **maximize** (*bool**, **optional*) -- maximize the params
    based on the objective, instead of minimizing (default: False)
differentiable (bool, optional) -- whether autograd
        should occur through the optimizer step in training.
        Otherwise, the step() function runs in a torch.no_grad()
        context. Setting to True can impair performance, so leave it
        False if you don't intend to run autograd through this
        instance (default: False)

add_param_group(param_group)
  Add a param group to the "Optimizer" s *param_groups*.

  This can be useful when fine tuning a pre-trained network as
  frozen layers can be made trainable and added to the "Optimizer"
  as training progresses.

  Parameters:
     **param_group** (*dict*) -- Specifies what Tensors should be
     optimized along with group specific optimization options.

load_state_dict(state_dict)
  Loads the optimizer state.

  Parameters:
     **state_dict** (*dict*) -- optimizer state. Should be an
     object returned from a call to "state_dict()".

register_step_post_hook(hook)
register_step_post_hook(hook)
  Register an optimizer step post hook which will be called after
  optimizer step. It should have the following signature:

     hook(optimizer, args, kwargs) -> None

  The "optimizer" argument is the optimizer instance being used.

  Parameters:
     **hook** (*Callable*) -- The user defined hook to be
     registered.

  Returns:
     a handle that can be used to remove the added hook by calling
     "handle.remove()"

  Return type:
     "torch.utils.hooks.RemoveableHandle"

register_step_pre_hook(hook)
  Register an optimizer step pre hook which will be called before
  optimizer step. It should have the following signature:

     hook(optimizer, args, kwargs) -> None or modified args and kwargs

  The "optimizer" argument is the optimizer instance being used.
  If args and kwargs are modified by the pre-hook, then the
  transformed values are returned as a tuple containing the
new_args and new_kwargs.
  Parameters:
     **hook** (*Callable*) -- The user defined hook to be
     registered.

  Returns:
     a handle that can be used to remove the added hook by calling
     "handle.remove()"

  Return type:
     "torch.utils.hooks.RemoveableHandle"

state_dict()
  Returns the state of the optimizer as a "dict".

  It contains two entries:

  * state - a dict holding current optimization state. Its content
       differs between optimizer classes.

  * param_groups - a list containing all parameter groups where
    each
       parameter group is a dict

zero_grad(set_to_none=False)
  Sets the gradients of all optimized "torch.Tensor" s to zero.

  Parameters:
     **set_to_none** (*bool*) -- instead of setting to zero, set
     the grads to None. This will in general have lower memory
     footprint, and can modestly improve performance. However, it
changes certain behaviors. For example: 1. When the user
         tries to access a gradient and perform manual ops on it, a
         None attribute or a Tensor full of 0s will behave
         differently. 2. If the user requests
         "zero_grad(set_to_none=True)" followed by a backward pass,
         ".grad"s are guaranteed to be None for params that did not
         receive a gradient. 3. "torch.optim" optimizers have a
         different behavior if the gradient is 0 or None (in one case
         it does the step with a gradient of 0 and in the other it
         skips the step altogether).
torch.qr
torch.qr(input, some=True, *, out=None)
Computes the QR decomposition of a matrix or a batch of matrices
   "input", and returns a namedtuple (Q, R) of tensors such that
   \text{input} = Q R with Q being an orthogonal matrix or batch of
   orthogonal matrices and R being an upper triangular matrix or batch
   of upper triangular matrices.
If "some" is "True", then this function returns the thin (reduced)
   QR factorization. Otherwise, if "some" is "False", this function
   returns the complete QR factorization.
Warning:
 "torch.qr()" is deprecated in favor of "torch.linalg.qr()" and
 will be removed in a future PyTorch release. The boolean
 parameter "some" has been replaced with a string parameter
 "mode"."Q, R = torch.qr(A)" should be replaced with

    Q, R = torch.linalg.qr(A)

 "Q, R = torch.qr(A, some=False)" should be replaced with

    Q, R = torch.linalg.qr(A, mode="complete")

Warning:
Warning:
 If you plan to backpropagate through QR, note that the current
 backward implementation is only well-defined when the first
 \min(input.size(-1), input.size(-2)) columns of "input" are
 linearly independent. This behavior will probably change once QR
 supports pivoting.

Note:
 This function uses LAPACK for CPU inputs and MAGMA for CUDA
 inputs, and may produce different (valid) decompositions on
 different device types or different platforms.

Parameters:
      * input (Tensor) -- the input tensor of size (*, m, n)
        where *** is zero or more batch dimensions consisting of
        matrices of dimension m \times n.
  * **some** (*bool**, **optional*) --

    Set to "True" for reduced QR decomposition and "False" for
    complete QR decomposition. If *k = min(m, n)* then:

       * "some=True" : returns *(Q, R)* with dimensions (m, k),
         (k, n) (default)
(k, n) (default)
       * "'some=False'": returns *(Q, R)* with dimensions (m, m),
         (m, n)

Keyword Arguments:
      out (tuple, optional) -- tuple of Q and R tensors.
      The dimensions of Q and R are detailed in the description of
      "some" above.
Example:
  >>> a = torch.tensor([[12., -51, 4], [6, 167, -68], [-4, 24, -41]])
  >>> q, r = torch.qr(a)
  >>> q
  tensor([[-0.8571,  0.3943,  0.3314],
          [-0.4286, -0.9029, -0.0343],
          [ 0.2857, -0.1714,  0.9429]])
  >>> r
  tensor([[ -14.0000,  -21.0000,   14.0000],
          [   0.0000, -175.0000,   70.0000],
          [   0.0000,    0.0000,  -35.0000]])
  >>> torch.mm(q, r).round()
  tensor([[  12.,  -51.,    4.],
          [   6.,  167.,  -68.],
          [  -4.,   24.,  -41.]])
  >>> torch.mm(q.t(), q).round()
  tensor([[ 1.,  0.,  0.],
          [ 0.,  1., -0.],
          [ 0., -0.,  1.]])
torch.linalg.lu_factor_ex
torch.linalg.lu_factor_ex(A, *, pivot=True, check_errors=False, out=None)
This is a version of "lu_factor()" that does not perform error
   checks unless "check_errors"= True. It also returns the "info"
   tensor returned by LAPACK's getrf.
Note:
 When the inputs are on a CUDA device, this function synchronizes
 only when "check_errors"*= True*.

Warning:
 This function is "experimental" and it may change in a future
 PyTorch release.

Parameters:
      A (Tensor) -- tensor of shape (, m, n)* where *** is
      zero or more batch dimensions.
Keyword Arguments:
      * pivot (bool, optional) -- Whether to compute the LU
        decomposition with partial pivoting, or the regular LU
        decomposition. "pivot"= False not supported on CPU. Default:
        True.
  * **check_errors** (*bool**, **optional*) -- controls whether to
torch.logcumsumexp
torch.logcumsumexp(input, dim, *, out=None) -> Tensor
Returns the logarithm of the cumulative summation of the
   exponentiation of elements of "input" in the dimension "dim".
For summation index j given by dim and other indices i, the
   result is
     \text{logcumsumexp}(x)_{ij} = \log \sum\limits_{j=0}^{i}
     \exp(x_{ij})

Parameters:
      * input (Tensor) -- the input tensor.
  * **dim** (*int*) -- the dimension to do the operation over

Keyword Arguments:
      out (Tensor, optional) -- the output tensor.
Example:
  >>> a = torch.randn(10)
  >>> torch.logcumsumexp(a, dim=0)
  tensor([-0.42296738, -0.04462666,  0.86278635,  0.94622083,  1.05277811,
           1.39202815,  1.83525007,  1.84492621,  2.06084887,  2.06844475]))
torch.diagonal
torch.diagonal(input, offset=0, dim1=0, dim2=1) -> Tensor
Returns a partial view of "input" with the its diagonal elements
   with respect to "dim1" and "dim2" appended as a dimension at the
   end of the shape.
The argument "offset" controls which diagonal to consider:


If "offset" = 0, it is the main diagonal.


If "offset" > 0, it is above the main diagonal.


If "offset" < 0, it is below the main diagonal.


Applying "torch.diag_embed()" to the output of this function with
   the same arguments yields a diagonal matrix with the diagonal
   entries of the input. However, "torch.diag_embed()" has different
   default dimensions, so those need to be explicitly specified.
Parameters:
      * input (Tensor) -- the input tensor. Must be at least
        2-dimensional.
  * **offset** (*int**, **optional*) -- which diagonal to
    consider. Default: 0 (main diagonal).

  * **dim1** (*int**, **optional*) -- first dimension with respect