shrinath-suresh/blogs-docs-splitted huggingface dataset:
CUDAGraph
class torch.cuda.CUDAGraph
Wrapper around a CUDA graph.
Warning:
 This API is in beta and may change in future releases.

capture_begin(pool=None)
  Begins capturing CUDA work on the current stream.

  Typically, you shouldn't call "capture_begin" yourself. Use
  "graph" or "make_graphed_callables()", which call
  "capture_begin" internally.

  Parameters:
     **pool** (*optional*) -- Token (returned by
     "graph_pool_handle()" or "other_Graph_instance.pool()") that
     hints this graph may share memory with the indicated pool.
     See Graph memory management.

capture_end()
  Ends CUDA graph capture on the current stream. After
  "capture_end", "replay" may be called on this instance.

  Typically, you shouldn't call "capture_end" yourself. Use
  "graph" or "make_graphed_callables()", which call "capture_end"
  internally.

debug_dump(debug_path)
  Parameters:
debug_dump(debug_path)
  Parameters:
     **debug_path** (*required*) -- Path to dump the graph to.

  Calls a debugging function to dump the graph if the debugging is
  enabled via CUDAGraph.enable_debug_mode()

enable_debug_mode()
  Enables debugging mode for CUDAGraph.debug_dump.

pool()
  Returns an opaque token representing the id of this graph's
  memory pool. This id can optionally be passed to another graph's
  "capture_begin", which hints the other graph may share the same
  memory pool.

replay()
  Replays the CUDA work captured by this graph.

reset()
  Deletes the graph currently held by this instance.
Softmax2d
class torch.nn.Softmax2d
Applies SoftMax over features to each spatial location.
When given an image of "Channels x Height x Width", it will apply
   Softmax to each location (Channels, h_i, w_j)
Shape:
      * Input: (N, C, H, W) or (C, H, W).
  * Output: (N, C, H, W) or (C, H, W) (same shape as input)

Returns:
      a Tensor of the same dimension and shape as the input with
      values in the range [0, 1]
Return type:
      None
Examples:
  >>> m = nn.Softmax2d()
  >>> # you softmax over the 2nd dimension
  >>> input = torch.randn(2, 3, 12, 13)
  >>> output = m(input)
torch.chunk
torch.chunk(input, chunks, dim=0) -> List of Tensors
Attempts to split a tensor into the specified number of chunks.
   Each chunk is a view of the input tensor.
Note:
 This function may return fewer than the specified number of
 chunks!

See also:
 "torch.tensor_split()" a function that always returns exactly the
 specified number of chunks

If the tensor size along the given dimension "dim" is divisible by
   "chunks", all returned chunks will be the same size. If the tensor
   size along the given dimension "dim" is not divisible by "chunks",
   all returned chunks will be the same size, except the last one. If
   such division is not possible, this function may return fewer than
   the specified number of chunks.
Parameters:
      * input (Tensor) -- the tensor to split
  * **chunks** (*int*) -- number of chunks to return

  * **dim** (*int*) -- dimension along which to split the tensor

-[ Example ]-
-[ Example ]-



torch.arange(11).chunk(6)
   (tensor([0, 1]),
    tensor([2, 3]),
    tensor([4, 5]),
    tensor([6, 7]),
    tensor([8, 9]),
    tensor([10]))
torch.arange(12).chunk(6)
   (tensor([0, 1]),
    tensor([2, 3]),
    tensor([4, 5]),
    tensor([6, 7]),
    tensor([8, 9]),
    tensor([10, 11]))
torch.arange(13).chunk(6)
   (tensor([0, 1, 2]),
    tensor([3, 4, 5]),
    tensor([6, 7, 8]),
    tensor([ 9, 10, 11]),
    tensor([12]))
SGD
class torch.optim.SGD(params, lr=, momentum=0, dampening=0, weight_decay=0, nesterov=False, *, maximize=False, foreach=None, differentiable=False)
Implements stochastic gradient descent (optionally with momentum).
  \begin{aligned}      &\rule{110mm}{0.4pt}
  \\      &\textbf{input}      : \gamma \text{ (lr)}, \: \theta_0
  \text{ (params)}, \: f(\theta)          \text{ (objective)}, \:
  \lambda \text{ (weight decay)},                          \\
  &\hspace{13mm} \:\mu \text{ (momentum)}, \:\tau \text{
  (dampening)},      \:\textit{ nesterov,}\:\textit{ maximize}
  \\[-1.ex]      &\rule{110mm}{0.4pt}
  \\      &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \:
  \textbf{do}                         \\      &\hspace{5mm}g_t
  \leftarrow   \nabla_{\theta} f_t (\theta_{t-1})           \\
  &\hspace{5mm}\textbf{if} \: \lambda \neq 0
  \\      &\hspace{10mm} g_t \leftarrow g_t + \lambda
  \theta_{t-1}                            \\
\theta_{t-1}                            \
      &\hspace{5mm}\textbf{if} \: \mu \neq 0
      \      &\hspace{10mm}\textbf{if} \: t > 1
      \      &\hspace{15mm} \textbf{b}t \leftarrow \mu
      \textbf{b} + (1-\tau) g_t           \
      &\hspace{10mm}\textbf{else}
      \      &\hspace{15mm} \textbf{b}t \leftarrow g_t
      \      &\hspace{10mm}\textbf{if} \: \textit{nesterov}
      \      &\hspace{15mm} g_t \leftarrow g + \mu \textbf{b}t
      \      &\hspace{10mm}\textbf{else}
      \[-1.ex]      &\hspace{15mm} g_t  \leftarrow  \textbf{b}_t
      \      &\hspace{5mm}\textbf{if} \: \textit{maximize}
      \      &\hspace{10mm}\theta_t \leftarrow \theta + \gamma
      g_t                   \[-1.ex]      &\hspace{5mm}\textbf{else}
      \[-1.ex]      &\hspace{10mm}\theta_t \leftarrow \theta_{t-1} -
      \gamma g_t                   \[-1.ex]      &\rule{110mm}{0.4pt}
      \[-1.ex]      &\bf{return} \:  \theta_t
      \[-1.ex]      &\rule{110mm}{0.4pt}
\[-1.ex]      &\rule{110mm}{0.4pt}
      \[-1.ex] \end{aligned}
Nesterov momentum is based on the formula from On the importance of
   initialization and momentum in deep learning.
Parameters:
      * params (iterable) -- iterable of parameters to optimize
        or dicts defining parameter groups
  * **lr** (*float*) -- learning rate

  * **momentum** (*float**, **optional*) -- momentum factor
    (default: 0)

  * **weight_decay** (*float**, **optional*) -- weight decay (L2
    penalty) (default: 0)

  * **dampening** (*float**, **optional*) -- dampening for
    momentum (default: 0)

  * **nesterov** (*bool**, **optional*) -- enables Nesterov
    momentum (default: False)

  * **maximize** (*bool**, **optional*) -- maximize the params
    based on the objective, instead of minimizing (default: False)

  * **foreach** (*bool**, **optional*) -- whether foreach
    implementation of optimizer is used (default: None)
differentiable (bool, optional) -- whether autograd
        should occur through the optimizer step in training.
        Otherwise, the step() function runs in a torch.no_grad()
        context. Setting to True can impair performance, so leave it
        False if you don't intend to run autograd through this
        instance (default: False)

-[ Example ]-



optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
optimizer.zero_grad()
loss_fn(model(input), target).backward()
optimizer.step()



Note:
 The implementation of SGD with Momentum/Nesterov subtly differs
 from Sutskever et. al. and implementations in some other
 frameworks.Considering the specific case of Momentum, the update
 can be written as

    \begin{aligned}     v_{t+1} & = \mu * v_{t} + g_{t+1}, \\
    p_{t+1} & = p_{t} - \text{lr} * v_{t+1}, \end{aligned}

 where p, g, v and \mu denote the parameters, gradient, velocity,
and momentum respectively.This is in contrast to Sutskever et.
     al. and other frameworks which employ an update of the form
    \begin{aligned}     v_{t+1} & = \mu * v_{t} + \text{lr} *
    g_{t+1}, \\     p_{t+1} & = p_{t} - v_{t+1}. \end{aligned}

 The Nesterov version is analogously modified.Moreover, the
 initial value of the momentum buffer is set to the gradient value
 at the first step. This is in contrast to some other frameworks
 that initialize it to all zeros.

add_param_group(param_group)
  Add a param group to the "Optimizer" s *param_groups*.

  This can be useful when fine tuning a pre-trained network as
  frozen layers can be made trainable and added to the "Optimizer"
  as training progresses.

  Parameters:
     **param_group** (*dict*) -- Specifies what Tensors should be
     optimized along with group specific optimization options.

load_state_dict(state_dict)
  Loads the optimizer state.

  Parameters:
Parameters:
         state_dict (dict) -- optimizer state. Should be an
         object returned from a call to "state_dict()".
register_step_post_hook(hook)
  Register an optimizer step post hook which will be called after
  optimizer step. It should have the following signature:

     hook(optimizer, args, kwargs) -> None

  The "optimizer" argument is the optimizer instance being used.

  Parameters:
     **hook** (*Callable*) -- The user defined hook to be
     registered.

  Returns:
     a handle that can be used to remove the added hook by calling
     "handle.remove()"

  Return type:
     "torch.utils.hooks.RemoveableHandle"

register_step_pre_hook(hook)
  Register an optimizer step pre hook which will be called before
  optimizer step. It should have the following signature:

     hook(optimizer, args, kwargs) -> None or modified args and kwargs

  The "optimizer" argument is the optimizer instance being used.
If args and kwargs are modified by the pre-hook, then the
      transformed values are returned as a tuple containing the
      new_args and new_kwargs.
  Parameters:
     **hook** (*Callable*) -- The user defined hook to be
     registered.

  Returns:
     a handle that can be used to remove the added hook by calling
     "handle.remove()"

  Return type:
     "torch.utils.hooks.RemoveableHandle"

state_dict()
  Returns the state of the optimizer as a "dict".

  It contains two entries:

  * state - a dict holding current optimization state. Its content
       differs between optimizer classes.

  * param_groups - a list containing all parameter groups where
    each
       parameter group is a dict

zero_grad(set_to_none=False)
  Sets the gradients of all optimized "torch.Tensor" s to zero.

  Parameters:
     **set_to_none** (*bool*) -- instead of setting to zero, set
the grads to None. This will in general have lower memory
         footprint, and can modestly improve performance. However, it
         changes certain behaviors. For example: 1. When the user
         tries to access a gradient and perform manual ops on it, a
         None attribute or a Tensor full of 0s will behave
         differently. 2. If the user requests
         "zero_grad(set_to_none=True)" followed by a backward pass,
         ".grad"s are guaranteed to be None for params that did not
         receive a gradient. 3. "torch.optim" optimizers have a
         different behavior if the gradient is 0 or None (in one case
         it does the step with a gradient of 0 and in the other it
         skips the step altogether).
torch.std
torch.std(input, dim=None, *, correction=1, keepdim=False, out=None) -> Tensor
Calculates the standard deviation over the dimensions specified by
   "dim". "dim" can be a single dimension, list of dimensions, or
   "None" to reduce over all dimensions.
The standard deviation (\sigma) is calculated as
  \sigma = \sqrt{\frac{1}{N - \delta
  N}\sum_{i=0}^{N-1}(x_i-\bar{x})^2}

where x is the sample set of elements, \bar{x} is the sample mean,
   N is the number of samples and \delta N is the "correction".
If "keepdim" is "True", the output tensor is of the same size as
   "input" except in the dimension(s) "dim" where it is of size 1.
   Otherwise, "dim" is squeezed (see "torch.squeeze()"), resulting in
   the output tensor having 1 (or "len(dim)") fewer dimension(s).
Parameters:
      * input (Tensor) -- the input tensor.
  * **dim** (*int** or **tuple of ints*) -- the dimension or
    dimensions to reduce.

Keyword Arguments:
Keyword Arguments:
      * correction (int) --
    difference between the sample size and sample degrees of
    freedom. Defaults to Bessel's correction, "correction=1".

    Changed in version 2.0: Previously this argument was called
    "unbiased" and was a boolean with "True" corresponding to
    "correction=1" and "False" being "correction=0".

  * **keepdim** (*bool*) -- whether the output tensor has "dim"
    retained or not.

  * **out** (*Tensor**, **optional*) -- the output tensor.

-[ Example ]-



a = torch.tensor(
   ...     [[ 0.2035,  1.2959,  1.8101, -0.4644],
   ...      [ 1.5027, -0.3270,  0.5905,  0.6538],
   ...      [-1.5745,  1.3330, -0.5596, -0.6548],
   ...      [ 0.1264, -0.5080,  1.6420,  0.1992]])
torch.std(a, dim=1, keepdim=True)
   tensor([[1.0311],
           [0.7477],
           [1.2204],
           [0.9087]])
torch.linalg.inv
torch.linalg.inv(A, *, out=None) -> Tensor
Computes the inverse of a square matrix if it exists. Throws a
   RuntimeError if the matrix is not invertible.
Letting \mathbb{K} be \mathbb{R} or \mathbb{C}, for a matrix A \in
   \mathbb{K}^{n \times n}, its inverse matrix A^{-1} \in
   \mathbb{K}^{n \times n} (if it exists) is defined as
  A^{-1}A = AA^{-1} = \mathrm{I}_n

where \mathrm{I}_n is the n-dimensional identity matrix.
The inverse matrix exists if and only if A is invertible. In this
   case, the inverse is unique.
Supports input of float, double, cfloat and cdouble dtypes. Also
   supports batches of matrices, and if "A" is a batch of matrices
   then the output has the same batch dimensions.
Note:
 When inputs are on a CUDA device, this function synchronizes that
 device with the CPU.

Note:
 Consider using "torch.linalg.solve()" if possible for multiplying
 a matrix on the left by the inverse, as:
a matrix on the left by the inverse, as:
    linalg.solve(A, B) == linalg.inv(A) @ B  # When B is a matrix

 It is always preferred to use "solve()" when possible, as it is
 faster and more numerically stable than computing the inverse
 explicitly.

See also:
 "torch.linalg.pinv()" computes the pseudoinverse (Moore-Penrose
 inverse) of matrices of any shape.

 "torch.linalg.solve()" computes "A"*.inv() @ *"B" with a
 numerically stable algorithm.

Parameters:
      A (Tensor) -- tensor of shape (, n, n)* where *** is
      zero or more batch dimensions consisting of invertible matrices.
Keyword Arguments:
      out (Tensor, optional) -- output tensor. Ignored if
      None. Default: None.
Raises:
      RuntimeError -- if the matrix "A" or any matrix in the batch
      of matrices "A" is not invertible.
Examples:
  >>> A = torch.randn(4, 4)
  >>> Ainv = torch.linalg.inv(A)
  >>> torch.dist(A @ Ainv, torch.eye(4))
torch.dist(A @ Ainv, torch.eye(4))
      tensor(1.1921e-07)



  >>> A = torch.randn(2, 3, 4, 4)  # Batch of matrices
  >>> Ainv = torch.linalg.inv(A)
  >>> torch.dist(A @ Ainv, torch.eye(4))
  tensor(1.9073e-06)

  >>> A = torch.randn(4, 4, dtype=torch.complex128)  # Complex matrix
  >>> Ainv = torch.linalg.inv(A)
  >>> torch.dist(A @ Ainv, torch.eye(4))
  tensor(7.5107e-16, dtype=torch.float64)
ChannelShuffle
class torch.nn.ChannelShuffle(groups)
Divide the channels in a tensor of shape (, C , H, W) into g
   groups and rearrange them as (, C \frac g, g, H, W), while keeping
   the original tensor shape.
Parameters:
      groups (int) -- number of groups to divide channels in.
Examples:
  >>> channel_shuffle = nn.ChannelShuffle(2)
  >>> input = torch.randn(1, 4, 2, 2)
  >>> print(input)
  [[[[1, 2],
     [3, 4]],
    [[5, 6],
     [7, 8]],
    [[9, 10],
     [11, 12]],
    [[13, 14],
     [15, 16]],
   ]]
  >>> output = channel_shuffle(input)
  >>> print(output)
  [[[[1, 2],
     [3, 4]],
    [[9, 10],
     [11, 12]],
    [[5, 6],
     [7, 8]],
    [[13, 14],
     [15, 16]],
   ]]
torch.gt
torch.gt(input, other, *, out=None) -> Tensor
Computes \text{input} > \text{other} element-wise.
The second argument can be a number or a tensor whose shape is
   broadcastable with the first argument.
Parameters:
      * input (Tensor) -- the tensor to compare
  * **other** (*Tensor** or **float*) -- the tensor or value to
    compare

Keyword Arguments:
      out (Tensor, optional) -- the output tensor.
Returns:
      A boolean tensor that is True where "input" is greater than
      "other" and False elsewhere
Example:
  >>> torch.gt(torch.tensor([[1, 2], [3, 4]]), torch.tensor([[1, 1], [4, 4]]))
  tensor([[False, True], [False, False]])
Bilinear
class torch.nn.Bilinear(in1_features, in2_features, out_features, bias=True, device=None, dtype=None)
Applies a bilinear transformation to the incoming data: y = x_1^T A
   x_2 + b
Parameters:
      * in1_features (int) -- size of each first input sample
  * **in2_features** (*int*) -- size of each second input sample

  * **out_features** (*int*) -- size of each output sample

  * **bias** (*bool*) -- If set to False, the layer will not learn
    an additive bias. Default: "True"

Shape:
      * Input1: (*, H_{in1}) where H_{in1}=\text{in1_features} and *
        means any number of additional dimensions including none. All
        but the last dimension of the inputs should be the same.
  * Input2: (*, H_{in2}) where H_{in2}=\text{in2\_features}.

  * Output: (*, H_{out}) where H_{out}=\text{out\_features} and
    all but the last dimension are the same shape as the input.

Variables:
Variables:
      * weight (torch.Tensor) -- the learnable weights of the
        module of shape (\text{out_features}, \text{in1_features},
        \text{in2_features}). The values are initialized from
        \mathcal{U}(-\sqrt{k}, \sqrt{k}), where k =
        \frac{1}{\text{in1_features}}
  * **bias** -- the learnable bias of the module of shape
    (\text{out\_features}). If "bias" is "True", the values are
    initialized from \mathcal{U}(-\sqrt{k}, \sqrt{k}), where k =
    \frac{1}{\text{in1\_features}}

Examples:
  >>> m = nn.Bilinear(20, 30, 40)
  >>> input1 = torch.randn(128, 20)
  >>> input2 = torch.randn(128, 30)
  >>> output = m(input1, input2)
  >>> print(output.size())
  torch.Size([128, 40])
torch.arange
torch.arange(start=0, end, step=1, *, out=None, dtype=None, layout=torch.strided, device=None, requires_grad=False) -> Tensor
Returns a 1-D tensor of size \left\lceil \frac{\text{end} -
   \text{start}}{\text{step}} \right\rceil with values from the
   interval "[start, end)" taken with common difference "step"
   beginning from start.
Note that non-integer "step" is subject to floating point rounding
   errors when comparing against "end"; to avoid inconsistency, we
   advise adding a small epsilon to "end" in such cases.
  \text{out}_{{i+1}} = \text{out}_{i} + \text{step}

Parameters:
      * start (Number) -- the starting value for the set of
        points. Default: "0".
  * **end** (*Number*) -- the ending value for the set of points

  * **step** (*Number*) -- the gap between each pair of adjacent
    points. Default: "1".

Keyword Arguments:
      * out (Tensor, optional) -- the output tensor.
dtype ("torch.dtype", optional) -- the desired data type
        of returned tensor. Default: if "None", uses a global default
        (see "torch.set_default_tensor_type()"). If dtype is not
        given, infer the data type from the other input arguments. If
        any of start, end, or stop are floating-point, the
        dtype is inferred to be the default dtype, see
        "get_default_dtype()". Otherwise, the dtype is inferred to
        be torch.int64.


layout ("torch.layout", optional) -- the desired layout of
    returned Tensor. Default: "torch.strided".


device ("torch.device", optional) -- the desired device of
    returned tensor. Default: if "None", uses the current device
    for the default tensor type (see
    "torch.set_default_tensor_type()"). "device" will be the CPU
    for CPU tensor types and the current CUDA device for CUDA
    tensor types.
tensor types.
  * **requires_grad** (*bool**, **optional*) -- If autograd should
    record operations on the returned tensor. Default: "False".

Example:
  >>> torch.arange(5)
  tensor([ 0,  1,  2,  3,  4])
  >>> torch.arange(1, 4)
  tensor([ 1,  2,  3])
  >>> torch.arange(1, 2.5, 0.5)
  tensor([ 1.0000,  1.5000,  2.0000])
torch.signal.windows.hann
torch.signal.windows.hann(M, *, sym=True, dtype=None, layout=torch.strided, device=None, requires_grad=False)
Computes the Hann window.
The Hann window is defined as follows:
  w_n = \frac{1}{2}\ \left[1 - \cos \left( \frac{2 \pi n}{M - 1}
  \right)\right] = \sin^2 \left( \frac{\pi n}{M - 1} \right)

The window is normalized to 1 (maximum value is 1). However, the 1
   doesn't appear if "M" is even and "sym" is True.
Parameters:
      M (int) -- the length of the window. In other words, the
      number of points of the returned window.
Keyword Arguments:
      * sym (bool, optional) -- If False, returns a
        periodic window suitable for use in spectral analysis. If
        True, returns a symmetric window suitable for use in filter
        design. Default: True.
  * **dtype** ("torch.dtype", optional) -- the desired data type
    of returned tensor. Default: if "None", uses a global default
(see "torch.set_default_tensor_type()").
  * **layout** ("torch.layout", optional) -- the desired layout of
    returned Tensor. Default: "torch.strided".

  * **device** ("torch.device", optional) -- the desired device of
    returned tensor. Default: if "None", uses the current device
    for the default tensor type (see
    "torch.set_default_tensor_type()"). "device" will be the CPU
    for CPU tensor types and the current CUDA device for CUDA
    tensor types.

  * **requires_grad** (*bool**, **optional*) -- If autograd should
    record operations on the returned tensor. Default: "False".

Return type:
      Tensor
Examples:
  >>> # Generates a symmetric Hann window.
  >>> torch.signal.windows.hann(10)
  tensor([0.0000, 0.1170, 0.4132, 0.7500, 0.9698, 0.9698, 0.7500, 0.4132, 0.1170, 0.0000])

  >>> # Generates a periodic Hann window.
  >>> torch.signal.windows.hann(10, sym=False)
torch.maximum
torch.maximum(input, other, *, out=None) -> Tensor
Computes the element-wise maximum of "input" and "other".
Note:
 If one of the elements being compared is a NaN, then that element
 is returned. "maximum()" is not supported for tensors with
 complex dtypes.

Parameters:
      * input (Tensor) -- the input tensor.
  * **other** (*Tensor*) -- the second input tensor

Keyword Arguments:
      out (Tensor, optional) -- the output tensor.
Example:
  >>> a = torch.tensor((1, 2, -1))
  >>> b = torch.tensor((3, 0, 4))
  >>> torch.maximum(a, b)
  tensor([3, 2, 4])
Linear
class torch.nn.Linear(in_features, out_features, bias=True, device=None, dtype=None)
Applies a linear transformation to the incoming data: y = xA^T + b
This module supports TensorFloat32.
On certain ROCm devices, when using float16 inputs this module will
   use different precision for backward.
Parameters:
      * in_features (int) -- size of each input sample
  * **out_features** (*int*) -- size of each output sample

  * **bias** (*bool*) -- If set to "False", the layer will not
    learn an additive bias. Default: "True"

Shape:
      * Input: (*, H_{in}) where * means any number of dimensions
        including none and H_{in} = \text{in_features}.
  * Output: (*, H_{out}) where all but the last dimension are the
    same shape as the input and H_{out} = \text{out\_features}.

Variables:
      * weight (torch.Tensor) -- the learnable weights of the
        module of shape (\text{out_features}, \text{in_features}).
The values are initialized from \mathcal{U}(-\sqrt{k},
        \sqrt{k}), where k = \frac{1}{\text{in_features}}
  * **bias** -- the learnable bias of the module of shape
    (\text{out\_features}). If "bias" is "True", the values are
    initialized from \mathcal{U}(-\sqrt{k}, \sqrt{k}) where k =
    \frac{1}{\text{in\_features}}

Examples:
  >>> m = nn.Linear(20, 30)
  >>> input = torch.randn(128, 20)
  >>> output = m(input)
  >>> print(output.size())
  torch.Size([128, 30])