shrinath-suresh/blogs-docs-splitted huggingface dataset:
Default: ""weight"".
  * **n_power_iterations** (*int**, **optional*) -- number of
    power iterations to calculate spectral norm. Default: "1".

  * **eps** (*float**, **optional*) -- epsilon for numerical
    stability in calculating norms. Default: "1e-12".

  * **dim** (*int**, **optional*) -- dimension corresponding to
    number of outputs. Default: "0", except for modules that are
    instances of ConvTranspose{1,2,3}d, when it is "1"

Returns:
      The original module with a new parametrization registered to the
      specified weight
Return type:
      Module
Example:
  >>> snm = spectral_norm(nn.Linear(20, 40))
  >>> snm
  ParametrizedLinear(
    in_features=20, out_features=40, bias=True
    (parametrizations): ModuleDict(
      (weight): ParametrizationList(
        (0): _SpectralNorm()
      )
    )
  )
  >>> torch.linalg.matrix_norm(snm.weight, 2)
  tensor(1.0081, grad_fn=<AmaxBackward0>)
torch.sparse.spdiags
torch.sparse.spdiags(diagonals, offsets, shape, layout=None) -> Tensor
Creates a sparse 2D tensor by placing the values from rows of
   "diagonals" along specified diagonals of the output
The "offsets" tensor controls which diagonals are set.


If "offsets[i]" = 0, it is the main diagonal


If "offsets[i]" < 0, it is below the main diagonal


If "offsets[i]" > 0, it is above the main diagonal


The number of rows in "diagonals" must match the length of
   "offsets", and an offset may not be repeated.
Parameters:
      * diagonals (Tensor) -- Matrix storing diagonals row-wise
  * **offsets** (*Tensor*) -- The diagonals to be set, stored as a
    vector

  * **shape** (*2-tuple of ints*) -- The desired shape of the
    result

Keyword Arguments:
      layout ("torch.layout", optional) -- The desired layout of
      the returned tensor. "torch.sparse_coo", "torch.sparse_csc" and
"torch.sparse_csr" are supported. Default: "torch.sparse_coo"
Examples:
Set the main and first two lower diagonals of a matrix:
  >>> diags = torch.arange(9).reshape(3, 3)
  >>> diags
  tensor([[0, 1, 2],
          [3, 4, 5],
          [6, 7, 8]])
  >>> s = torch.sparse.spdiags(diags, torch.tensor([0, -1, -2]), (3, 3))
  >>> s
  tensor(indices=tensor([[0, 1, 2, 1, 2, 2],
                         [0, 1, 2, 0, 1, 0]]),
         values=tensor([0, 1, 2, 3, 4, 6]),
         size=(3, 3), nnz=6, layout=torch.sparse_coo)
  >>> s.to_dense()
  tensor([[0, 0, 0],
          [3, 1, 0],
          [6, 4, 2]])

Change the output layout:
  >>> diags = torch.arange(9).reshape(3, 3)
  >>> diags
  tensor([[0, 1, 2],[3, 4, 5], [6, 7, 8])
  >>> s = torch.sparse.spdiags(diags, torch.tensor([0, -1, -2]), (3, 3), layout=torch.sparse_csr)
  >>> s
  tensor(crow_indices=tensor([0, 1, 3, 6]),
tensor(crow_indices=tensor([0, 1, 3, 6]),
             col_indices=tensor([0, 0, 1, 0, 1, 2]),
             values=tensor([0, 3, 1, 6, 4, 2]), size=(3, 3), nnz=6,
             layout=torch.sparse_csr)
      >>> s.to_dense()
      tensor([[0, 0, 0],
              [3, 1, 0],
              [6, 4, 2]])
Set partial diagonals of a large output:
  >>> diags = torch.tensor([[1, 2], [3, 4]])
  >>> offsets = torch.tensor([0, -1])
  >>> torch.sparse.spdiags(diags, offsets, (5, 5)).to_dense()
  tensor([[1, 0, 0, 0, 0],
          [3, 2, 0, 0, 0],
          [0, 4, 0, 0, 0],
          [0, 0, 0, 0, 0],
          [0, 0, 0, 0, 0]])

Note:
 When setting the values along a given diagonal the index into the
 diagonal and the index into the row of "diagonals" is taken as
 the column index in the output. This has the effect that when
 setting a diagonal with a positive offset *k* the first value
 along that diagonal will be the value in position *k* of the row
of "diagonals"
Specifying a positive offset:
  >>> diags = torch.tensor([[1, 2, 3], [1, 2, 3], [1, 2, 3]])
  >>> torch.sparse.spdiags(diags, torch.tensor([0, 1, 2]), (5, 5)).to_dense()
  tensor([[1, 2, 3, 0, 0],
          [0, 2, 3, 0, 0],
          [0, 0, 3, 0, 0],
          [0, 0, 0, 0, 0],
          [0, 0, 0, 0, 0]])
ExternalStream
class torch.cuda.ExternalStream(stream_ptr, device=None, **kwargs)
Wrapper around an externally allocated CUDA stream.
This class is used to wrap streams allocated in other libraries in
   order to facilitate data exchange and multi-library interactions.
Note:
 This class doesn't manage the stream life-cycle, it is the user
 responsibility to keep the referenced stream alive while this
 class is being used.

Parameters:
      * stream_ptr (int) -- Integer representation of the
        cudaStream_t value. allocated externally.
  * **device** (*torch.device** or **int**, **optional*) -- the
    device where the stream was originally allocated. if device is
    specified incorrectly, subsequent launches using this stream
    may fail.

query()
  Checks if all the work submitted has been completed.

  Returns:
     A boolean indicating if all kernels in this stream are
     completed.
torch.tanh
torch.tanh(input, *, out=None) -> Tensor
Returns a new tensor with the hyperbolic tangent of the elements of
   "input".
  \text{out}_{i} = \tanh(\text{input}_{i})

Parameters:
      input (Tensor) -- the input tensor.
Keyword Arguments:
      out (Tensor, optional) -- the output tensor.
Example:
  >>> a = torch.randn(4)
  >>> a
  tensor([ 0.8986, -0.7279,  1.1745,  0.2611])
  >>> torch.tanh(a)
  tensor([ 0.7156, -0.6218,  0.8257,  0.2553])
torch.exp
torch.exp(input, *, out=None) -> Tensor
Returns a new tensor with the exponential of the elements of the
   input tensor "input".
  y_{i} = e^{x_{i}}

Parameters:
      input (Tensor) -- the input tensor.
Keyword Arguments:
      out (Tensor, optional) -- the output tensor.
Example:
  >>> torch.exp(torch.tensor([0, math.log(2.)]))
  tensor([ 1.,  2.])
Rprop
class torch.optim.Rprop(params, lr=0.01, etas=(0.5, 1.2), step_sizes=(1e-06, 50), *, foreach=None, maximize=False, differentiable=False)
Implements the resilient backpropagation algorithm.
  \begin{aligned}      &\rule{110mm}{0.4pt}
  \\      &\textbf{input}      : \theta_0 \in \mathbf{R}^d \text{
  (params)},f(\theta)          \text{ (objective)},
  \\      &\hspace{13mm}      \eta_{+/-} \text{ (etaplus,
  etaminus)}, \Gamma_{max/min}          \text{ (step sizes)}
  \\      &\textbf{initialize} :   g^0_{prev} \leftarrow 0,
  \: \eta_0 \leftarrow \text{lr (learning rate)}
  \\      &\rule{110mm}{0.4pt}
  \\      &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \:
  \textbf{do}                         \\      &\hspace{5mm}g_t
  \leftarrow   \nabla_{\theta} f_t (\theta_{t-1})           \\
  &\hspace{5mm} \textbf{for} \text{  } i = 0, 1, \ldots, d-1 \:
  \mathbf{do}            \\      &\hspace{10mm}  \textbf{if} \:
g^i_{prev} g^i_t  > 0                               \
      &\hspace{15mm}  \eta^i_t \leftarrow \mathrm{min}(\eta^i_{t-1}
      \eta_{+},          \Gamma_{max})
      \      &\hspace{10mm}  \textbf{else if}  \:  g^i_{prev} g^i_t <
      0                           \      &\hspace{15mm}  \eta^i_t
      \leftarrow \mathrm{max}(\eta^i_{t-1} \eta_{-},
      \Gamma_{min})
      \      &\hspace{15mm}  g^i_t \leftarrow 0
      \      &\hspace{10mm}  \textbf{else}  \:
      \      &\hspace{15mm}  \eta^i_t \leftarrow \eta^i_{t-1}
      \      &\hspace{5mm}\theta_t \leftarrow \theta_{t-1}- \eta_t
      \mathrm{sign}(g_t)             \      &\hspace{5mm}g_{prev}
      \leftarrow  g_t
      \      &\rule{110mm}{0.4pt}
      \[-1.ex]      &\bf{return} \:  \theta_t
      \[-1.ex]      &\rule{110mm}{0.4pt}
      \[-1.ex] \end{aligned}
For further details regarding the algorithm we refer to the paper A
   Direct Adaptive Method for Faster Backpropagation Learning: The
   RPROP Algorithm.
Parameters:
RPROP Algorithm.
Parameters:
      * params (iterable) -- iterable of parameters to optimize
        or dicts defining parameter groups
  * **lr** (*float**, **optional*) -- learning rate (default:
    1e-2)

  * **etas** (*Tuple**[**float**, **float**]**, **optional*) --
    pair of (etaminus, etaplus), that are multiplicative increase
    and decrease factors (default: (0.5, 1.2))

  * **step_sizes** (*Tuple**[**float**, **float**]**, **optional*)
    -- a pair of minimal and maximal allowed step sizes (default:
    (1e-6, 50))

  * **foreach** (*bool**, **optional*) -- whether foreach
    implementation of optimizer is used. If unspecified by the
    user (so foreach is None), we will try to use foreach over the
    for-loop implementation on CUDA, since it is usually
    significantly more performant. (default: None)

  * **maximize** (*bool**, **optional*) -- maximize the params
based on the objective, instead of minimizing (default: False)
  * **differentiable** (*bool**, **optional*) -- whether autograd
    should occur through the optimizer step in training.
    Otherwise, the step() function runs in a torch.no_grad()
    context. Setting to True can impair performance, so leave it
    False if you don't intend to run autograd through this
    instance (default: False)

add_param_group(param_group)
  Add a param group to the "Optimizer" s *param_groups*.

  This can be useful when fine tuning a pre-trained network as
  frozen layers can be made trainable and added to the "Optimizer"
  as training progresses.

  Parameters:
     **param_group** (*dict*) -- Specifies what Tensors should be
     optimized along with group specific optimization options.

load_state_dict(state_dict)
  Loads the optimizer state.

  Parameters:
     **state_dict** (*dict*) -- optimizer state. Should be an
transformed values are returned as a tuple containing the
      new_args and new_kwargs.
  Parameters:
     **hook** (*Callable*) -- The user defined hook to be
     registered.

  Returns:
     a handle that can be used to remove the added hook by calling
     "handle.remove()"

  Return type:
     "torch.utils.hooks.RemoveableHandle"

state_dict()
  Returns the state of the optimizer as a "dict".

  It contains two entries:

  * state - a dict holding current optimization state. Its content
       differs between optimizer classes.

  * param_groups - a list containing all parameter groups where
    each
       parameter group is a dict

zero_grad(set_to_none=False)
  Sets the gradients of all optimized "torch.Tensor" s to zero.

  Parameters:
     **set_to_none** (*bool*) -- instead of setting to zero, set
     the grads to None. This will in general have lower memory
footprint, and can modestly improve performance. However, it
         changes certain behaviors. For example: 1. When the user
         tries to access a gradient and perform manual ops on it, a
         None attribute or a Tensor full of 0s will behave
         differently. 2. If the user requests
         "zero_grad(set_to_none=True)" followed by a backward pass,
         ".grad"s are guaranteed to be None for params that did not
         receive a gradient. 3. "torch.optim" optimizers have a
         different behavior if the gradient is 0 or None (in one case
         it does the step with a gradient of 0 and in the other it
         skips the step altogether).
torch.fft.irfftn
torch.fft.irfftn(input, s=None, dim=None, norm=None, *, out=None) -> Tensor
Computes the inverse of "rfftn()".
"input" is interpreted as a one-sided Hermitian signal in the
   Fourier domain, as produced by "rfftn()". By the Hermitian
   property, the output will be real-valued.
Note:
 Some input frequencies must be real-valued to satisfy the
 Hermitian property. In these cases the imaginary component will
 be ignored. For example, any imaginary component in the zero-
 frequency term cannot be represented in a real output and so will
 always be ignored.

Note:
 The correct interpretation of the Hermitian input depends on the
 length of the original data, as given by "s". This is because
 each input shape could correspond to either an odd or even length
 signal. By default, the signal is assumed to be even length and
 odd signals will not round-trip properly. So, it is recommended
to always pass the signal shape "s".
Note:
 Supports torch.half and torch.chalf on CUDA with GPU Architecture
 SM53 or greater. However it only supports powers of 2 signal
 length in every transformed dimensions. With default arguments,
 the size of last dimension should be (2^n + 1) as argument *s*
 defaults to even output size = 2 * (last_dim_size - 1)

Parameters:
      * input (Tensor) -- the input tensor
  * **s** (*Tuple**[**int**]**, **optional*) -- Signal size in the
    transformed dimensions. If given, each dimension "dim[i]" will
    either be zero-padded or trimmed to the length "s[i]" before
    computing the real FFT. If a length "-1" is specified, no
    padding is done in that dimension. Defaults to even output in
    the last dimension: "s[-1] = 2*(input.size(dim[-1]) - 1)".

  * **dim** (*Tuple**[**int**]**, **optional*) -- Dimensions to be
    transformed. The last dimension must be the half-Hermitian
compressed dimension. Default: all dimensions, or the last
        "len(s)" dimensions if "s" is given.
  * **norm** (*str**, **optional*) --

    Normalization mode. For the backward transform ("irfftn()"),
    these correspond to:

    * ""forward"" - no normalization

    * ""backward"" - normalize by "1/n"

    * ""ortho"" - normalize by "1/sqrt(n)" (making the real IFFT
      orthonormal)

    Where "n = prod(s)" is the logical IFFT size. Calling the
    forward transform ("rfftn()") with the same normalization mode
    will apply an overall normalization of "1/n" between the two
    transforms. This is required to make "irfftn()" the exact
    inverse.

    Default is ""backward"" (normalize by "1/n").

Keyword Arguments:
      out (Tensor, optional) -- the output tensor.
-[ Example ]-



t = torch.rand(10, 9)
T = torch.fft.rfftn(t)



Without specifying the output length to "irfft()", the output will
not round-trip properly because the input is odd-length in the last
   dimension:



torch.fft.irfftn(T).size()
   torch.Size([10, 8])



So, it is recommended to always pass the signal shape "s".



roundtrip = torch.fft.irfftn(T, t.size())
roundtrip.size()
   torch.Size([10, 9])
torch.testing.assert_close(roundtrip, t, check_stride=False)
torch.linalg.inv_ex
torch.linalg.inv_ex(A, *, check_errors=False, out=None)
Computes the inverse of a square matrix if it is invertible.
Returns a namedtuple "(inverse, info)". "inverse" contains the
   result of inverting "A" and "info" stores the LAPACK error codes.
If "A" is not an invertible matrix, or if it's a batch of matrices
   and one or more of them is not an invertible matrix, then "info"
   stores a positive integer for the corresponding matrix. The
   positive integer indicates the diagonal element of the LU
   decomposition of the input matrix that is exactly zero. "info"
   filled with zeros indicates that the inversion was successful. If
   "check_errors=True" and "info" contains positive integers, then a
   RuntimeError is thrown.
Supports input of float, double, cfloat and cdouble dtypes. Also
   supports batches of matrices, and if "A" is a batch of matrices
   then the output has the same batch dimensions.
Note:
Note:
 When the inputs are on a CUDA device, this function synchronizes
 only when "check_errors"*= True*.

Warning:
 This function is "experimental" and it may change in a future
 PyTorch release.

See also:
 "torch.linalg.inv()" is a NumPy compatible variant that always
 checks for errors.

Parameters:
      * A (Tensor) -- tensor of shape (, n, n)* where *** is
        zero or more batch dimensions consisting of square matrices.
  * **check_errors** (*bool**, **optional*) -- controls whether to
    check the content of "info". Default: *False*.

Keyword Arguments:
      out (tuple, optional) -- tuple of two tensors to write
      the output to. Ignored if None. Default: None.
Examples:
  >>> A = torch.randn(3, 3)
  >>> Ainv, info = torch.linalg.inv_ex(A)
  >>> torch.dist(torch.linalg.inv(A), Ainv)
  tensor(0.)
  >>> info
  tensor(0, dtype=torch.int32)
torch.nn.functional.binary_cross_entropy_with_logits
torch.nn.functional.binary_cross_entropy_with_logits(input, target, weight=None, size_average=None, reduce=None, reduction='mean', pos_weight=None)
Function that measures Binary Cross Entropy between target and
   input logits.
See "BCEWithLogitsLoss" for details.
Parameters:
      * input (Tensor) -- Tensor of arbitrary shape as
        unnormalized scores (often referred to as logits).
  * **target** (*Tensor*) -- Tensor of the same shape as input
    with values between 0 and 1

  * **weight** (*Tensor**, **optional*) -- a manual rescaling
    weight if provided it's repeated to match input tensor shape

  * **size_average** (*bool**, **optional*) -- Deprecated (see
    "reduction"). By default, the losses are averaged over each
    loss element in the batch. Note that for some losses, there
multiple elements per sample. If the field "size_average" is
        set to "False", the losses are instead summed for each
        minibatch. Ignored when reduce is "False". Default: "True"
  * **reduce** (*bool**, **optional*) -- Deprecated (see
    "reduction"). By default, the losses are averaged or summed
    over observations for each minibatch depending on
    "size_average". When "reduce" is "False", returns a loss per
    batch element instead and ignores "size_average". Default:
    "True"

  * **reduction** (*str**, **optional*) -- Specifies the reduction
    to apply to the output: "'none'" | "'mean'" | "'sum'".
    "'none'": no reduction will be applied, "'mean'": the sum of
    the output will be divided by the number of elements in the
    output, "'sum'": the output will be summed. Note:
    "size_average" and "reduce" are in the process of being
    deprecated, and in the meantime, specifying either of those
two args will override "reduction". Default: "'mean'"
  * **pos_weight** (*Tensor**, **optional*) -- a weight of
    positive examples. Must be a vector with length equal to the
    number of classes.

Return type:
      Tensor
Examples:
  >>> input = torch.randn(3, requires_grad=True)
  >>> target = torch.empty(3).random_(2)
  >>> loss = F.binary_cross_entropy_with_logits(input, target)
  >>> loss.backward()
CyclicLR
class torch.optim.lr_scheduler.CyclicLR(optimizer, base_lr, max_lr, step_size_up=2000, step_size_down=None, mode='triangular', gamma=1.0, scale_fn=None, scale_mode='cycle', cycle_momentum=True, base_momentum=0.8, max_momentum=0.9, last_epoch=- 1, verbose=False)
Sets the learning rate of each parameter group according to
   cyclical learning rate policy (CLR). The policy cycles the learning
   rate between two boundaries with a constant frequency, as detailed
   in the paper Cyclical Learning Rates for Training Neural Networks.
   The distance between the two boundaries can be scaled on a per-
   iteration or per-cycle basis.
Cyclical learning rate policy changes the learning rate after every
   batch. step should be called after a batch has been used for
   training.
This class has three built-in policies, as put forth in the paper:


"triangular": A basic triangular cycle without amplitude scaling.


"triangular2": A basic triangular cycle that scales initial
amplitude by half each cycle.

"exp_range": A cycle that scales initial amplitude by
     \text{gamma}^{\text{cycle iterations}} at each cycle iteration.

This implementation was adapted from the github repo:
   bckenstler/CLR
Parameters:
      * optimizer (Optimizer) -- Wrapped optimizer.
  * **base_lr** (*float** or **list*) -- Initial learning rate
    which is the lower boundary in the cycle for each parameter
    group.

  * **max_lr** (*float** or **list*) -- Upper learning rate
    boundaries in the cycle for each parameter group.
    Functionally, it defines the cycle amplitude (max_lr -
    base_lr). The lr at any cycle is the sum of base_lr and some
    scaling of the amplitude; therefore max_lr may not actually be
    reached depending on scaling function.

  * **step_size_up** (*int*) -- Number of training iterations in
    the increasing half of a cycle. Default: 2000
step_size_down (int) -- Number of training iterations in
        the decreasing half of a cycle. If step_size_down is None, it
        is set to step_size_up. Default: None


mode (str) -- One of {triangular, triangular2,
    exp_range}. Values correspond to policies detailed above. If
    scale_fn is not None, this argument is ignored. Default:
    'triangular'


gamma (float) -- Constant in 'exp_range' scaling
    function: gamma**(cycle iterations) Default: 1.0


scale_fn (function) -- Custom scaling policy defined by
    a single argument lambda function, where 0 <= scale_fn(x) <= 1
    for all x >= 0. If specified, then 'mode' is ignored. Default:
    None


scale_mode (str) -- {'cycle', 'iterations'}. Defines
    whether scale_fn is evaluated on cycle number or cycle
    iterations (training iterations since start of cycle).
    Default: 'cycle'
Default: 'cycle'
  * **cycle_momentum** (*bool*) -- If "True", momentum is cycled
    inversely to learning rate between 'base_momentum' and
    'max_momentum'. Default: True

  * **base_momentum** (*float** or **list*) -- Lower momentum
    boundaries in the cycle for each parameter group. Note that
    momentum is cycled inversely to learning rate; at the peak of
    a cycle, momentum is 'base_momentum' and learning rate is
    'max_lr'. Default: 0.8

  * **max_momentum** (*float** or **list*) -- Upper momentum
    boundaries in the cycle for each parameter group.
    Functionally, it defines the cycle amplitude (max_momentum -
    base_momentum). The momentum at any cycle is the difference of
    max_momentum and some scaling of the amplitude; therefore
    base_momentum may not actually be reached depending on scaling
    function. Note that momentum is cycled inversely to learning
rate; at the start of a cycle, momentum is 'max_momentum' and
        learning rate is 'base_lr' Default: 0.9
  * **last_epoch** (*int*) -- The index of the last batch. This
    parameter is used when resuming a training job. Since *step()*
    should be invoked after each batch instead of after each
    epoch, this number represents the total number of *batches*
    computed, not the total number of epochs computed. When
    last_epoch=-1, the schedule is started from the beginning.
    Default: -1

  * **verbose** (*bool*) -- If "True", prints a message to stdout
    for each update. Default: "False".

-[ Example ]-



optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
scheduler = torch.optim.lr_scheduler.CyclicLR(optimizer, base_lr=0.01, max_lr=0.1)
data_loader = torch.utils.data.DataLoader(...)
for epoch in range(10):
    for batch in data_loader:
        train_batch(...)
train_batch(...)
    scheduler.step()




get_last_lr()
  Return last computed learning rate by current scheduler.

get_lr()
  Calculates the learning rate at batch index. This function
  treats *self.last_epoch* as the last batch index.

  If *self.cycle_momentum* is "True", this function has a side
  effect of updating the optimizer's momentum.

print_lr(is_verbose, group, lr, epoch=None)
  Display the current learning rate.
torch.onnx diagnostics
* Overview


Diagnostic Rules


API Reference


Overview
NOTE: This feature is underdevelopment and is subject to change.
The goal is to improve the diagnostics to help users debug and improve
their model export to ONNX.


The diagnostics are emitted in machine parsable Static Analysis
  Results Interchange Format (SARIF).


A new clearer, structured way to add new and keep track of
  diagnostic rules.


Serve as foundation for more future improvements consuming the
  diagnostics.


Diagnostic Rules


POE0001:node-missing-onnx-shape-inference


POE0002:missing-custom-symbolic-function


POE0003:missing-standard-symbolic-function


POE0004:operator-supported-in-newer-opset-version


API Reference
class torch.onnx._internal.diagnostics.ExportDiagnostic(args, *kwargs)
Base class for all export diagnostics.
This class is used to represent all export diagnostics. It is a