shrinath-suresh/blogs-docs-splitted huggingface dataset:
parameters and buffers in this module
     * **dtype** ("torch.dtype") -- the desired floating point or
       complex dtype of the parameters and buffers in this module

     * **tensor** (*torch.Tensor*) -- Tensor whose dtype and
       device are the desired dtype and device for all parameters
       and buffers in this module

     * **memory_format** ("torch.memory_format") -- the desired
       memory format for 4D parameters and buffers in this module
       (keyword only argument)

  Returns:
     self

  Return type:
     Module

  Examples:

     >>> linear = nn.Linear(2, 2)
     >>> linear.weight
     Parameter containing:
     tensor([[ 0.1913, -0.3420],
             [-0.5113, -0.2325]])
     >>> linear.to(torch.double)
     Linear(in_features=2, out_features=2, bias=True)
     >>> linear.weight
     Parameter containing:
     tensor([[ 0.1913, -0.3420],
tensor([[ 0.1913, -0.3420],
                 [-0.5113, -0.2325]], dtype=torch.float64)
         >>> gpu1 = torch.device("cuda:1")
         >>> linear.to(gpu1, dtype=torch.half, non_blocking=True)
         Linear(in_features=2, out_features=2, bias=True)
         >>> linear.weight
         Parameter containing:
         tensor([[ 0.1914, -0.3420],
                 [-0.5112, -0.2324]], dtype=torch.float16, device='cuda:1')
         >>> cpu = torch.device("cpu")
         >>> linear.to(cpu)
         Linear(in_features=2, out_features=2, bias=True)
         >>> linear.weight
         Parameter containing:
         tensor([[ 0.1914, -0.3420],
                 [-0.5112, -0.2324]], dtype=torch.float16)
     >>> linear = nn.Linear(2, 2, bias=None).to(torch.cdouble)
     >>> linear.weight
     Parameter containing:
     tensor([[ 0.3741+0.j,  0.2382+0.j],
             [ 0.5593+0.j, -0.4443+0.j]], dtype=torch.complex128)
     >>> linear(torch.ones(3, 2, dtype=torch.cdouble))
tensor([[0.6122+0.j, 0.1150+0.j],
                 [0.6122+0.j, 0.1150+0.j],
                 [0.6122+0.j, 0.1150+0.j]], dtype=torch.complex128)
to_empty(*, device)
  Moves the parameters and buffers to the specified device without
  copying storage.

  Parameters:
     **device** ("torch.device") -- The desired device of the
     parameters and buffers in this module.

  Returns:
     self

  Return type:
     Module

train(mode=True)
  Sets the module in training mode.

  This has any effect only on certain modules. See documentations
  of particular modules for details of their behaviors in
  training/evaluation mode, if they are affected, e.g. "Dropout",
  "BatchNorm", etc.

  Parameters:
     **mode** (*bool*) -- whether to set training mode ("True") or
     evaluation mode ("False"). Default: "True".

  Returns:
     self

  Return type:
     Module

type(dst_type)
Module
type(dst_type)
  Casts all parameters and buffers to "dst_type".

  Note:

    This method modifies the module in-place.

  Parameters:
     **dst_type** (*type** or **string*) -- the desired type

  Returns:
     self

  Return type:
     Module

xpu(device=None)
  Moves all model parameters and buffers to the XPU.

  This also makes associated parameters and buffers different
  objects. So it should be called before constructing optimizer if
  the module will live on XPU while being optimized.

  Note:

    This method modifies the module in-place.

  Parameters:
     **device** (*int**, **optional*) -- if specified, all
     parameters will be copied to that device

  Returns:
     self

  Return type:
     Module

zero_grad(set_to_none=False)
  Sets gradients of all model parameters to zero. See similar
  function under "torch.optim.Optimizer" for more context.

  Parameters:
torch.nn.functional.avg_pool2d
torch.nn.functional.avg_pool2d(input, kernel_size, stride=None, padding=0, ceil_mode=False, count_include_pad=True, divisor_override=None) -> Tensor
Applies 2D average-pooling operation in kH \times kW regions by
   step size sH \times sW steps. The number of output features is
   equal to the number of input planes.
See "AvgPool2d" for details and output shape.
Parameters:
      * input -- input tensor (\text{minibatch} ,
        \text{in_channels} , iH , iW)
  * **kernel_size** -- size of the pooling region. Can be a single
    number or a tuple *(kH, kW)*

  * **stride** -- stride of the pooling operation. Can be a single
    number or a tuple *(sH, sW)*. Default: "kernel_size"

  * **padding** -- implicit zero paddings on both sides of the
    input. Can be a single number or a tuple *(padH, padW)*.
    Default: 0

  * **ceil_mode** -- when True, will use *ceil* instead of *floor*
torch.Tensor.to_sparse
Tensor.to_sparse(sparseDims) -> Tensor
Returns a sparse copy of the tensor.  PyTorch supports sparse
   tensors in coordinate format.
Parameters:
      sparseDims (int, optional) -- the number of sparse
      dimensions to include in the new sparse tensor
Example:
  >>> d = torch.tensor([[0, 0, 0], [9, 0, 10], [0, 0, 0]])
  >>> d
  tensor([[ 0,  0,  0],
          [ 9,  0, 10],
          [ 0,  0,  0]])
  >>> d.to_sparse()
  tensor(indices=tensor([[1, 1],
                         [0, 2]]),
         values=tensor([ 9, 10]),
         size=(3, 3), nnz=2, layout=torch.sparse_coo)
  >>> d.to_sparse(1)
  tensor(indices=tensor([[1]]),
         values=tensor([[ 9,  0, 10]]),
         size=(3, 3), nnz=1, layout=torch.sparse_coo)

to_sparse(*, layout=None, blocksize=None, dense_dim=None) -> Tensor
Returns a sparse tensor with the specified layout and blocksize.
If the "self" is strided, the number of dense dimensions could be
   specified, and a hybrid sparse tensor will be created, with
   dense_dim dense dimensions and self.dim() - 2 - dense_dim batch
   dimension.
Note:
 If the "self" layout and blocksize parameters match with the
 specified layout and blocksize, return "self". Otherwise, return
 a sparse tensor copy of "self".

Parameters:
      * layout ("torch.layout", optional) -- The desired sparse
        layout. One of "torch.sparse_coo", "torch.sparse_csr",
        "torch.sparse_csc", "torch.sparse_bsr", or "torch.sparse_bsc".
        Default: if "None", "torch.sparse_coo".
  * **blocksize** (list, tuple, "torch.Size", optional) -- Block
    size of the resulting BSR or BSC tensor. For other layouts,
    specifying the block size that is not "None" will result in a
    RuntimeError exception.  A block size must be a tuple of
    length two such that its items evenly divide the two sparse
dimensions.
  * **dense_dim** (*int**, **optional*) -- Number of dense
    dimensions of the resulting CSR, CSC, BSR or BSC tensor.  This
    argument should be used only if "self" is a strided tensor,
    and must be a value between 0 and dimension of "self" tensor
    minus two.

Example:
  >>> x = torch.tensor([[1, 0], [0, 0], [2, 3]])
  >>> x.to_sparse(layout=torch.sparse_coo)
  tensor(indices=tensor([[0, 2, 2],
                         [0, 0, 1]]),
         values=tensor([1, 2, 3]),
         size=(3, 2), nnz=3, layout=torch.sparse_coo)
  >>> x.to_sparse(layout=torch.sparse_bsr, blocksize=(1, 2))
  tensor(crow_indices=tensor([0, 1, 1, 2]),
         col_indices=tensor([0, 0]),
         values=tensor([[[1, 0]],
                        [[2, 3]]]), size=(3, 2), nnz=2, layout=torch.sparse_bsr)
  >>> x.to_sparse(layout=torch.sparse_bsr, blocksize=(2, 1))
  RuntimeError: Tensor size(-2) 3 needs to be divisible by blocksize[0] 2
x.to_sparse(layout=torch.sparse_csr, blocksize=(3, 1))
      RuntimeError: to_sparse for Strided to SparseCsr conversion does not use specified blocksize



  >>> x = torch.tensor([[[1], [0]], [[0], [0]], [[2], [3]]])
  >>> x.to_sparse(layout=torch.sparse_csr, dense_dim=1)
  tensor(crow_indices=tensor([0, 1, 1, 3]),
         col_indices=tensor([0, 0, 1]),
         values=tensor([[1],
                        [2],
                        [3]]), size=(3, 2, 1), nnz=3, layout=torch.sparse_csr)
Adamax
class torch.optim.Adamax(params, lr=0.002, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, foreach=None, *, maximize=False, differentiable=False)
Implements Adamax algorithm (a variant of Adam based on infinity
   norm).
  \begin{aligned}      &\rule{110mm}{0.4pt}
  \\      &\textbf{input}      : \gamma \text{ (lr)}, \beta_1,
  \beta_2          \text{ (betas)},\theta_0 \text{
  (params)},f(\theta) \text{ (objective)},          \: \lambda
  \text{ (weight decay)},
  \\      &\hspace{13mm}    \epsilon \text{ (epsilon)}
  \\      &\textbf{initialize} :  m_0 \leftarrow 0 \text{ ( first
  moment)},          u_0 \leftarrow 0 \text{ ( infinity norm)}
  \\[-1.ex]      &\rule{110mm}{0.4pt}
  \\      &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \:
  \textbf{do}                         \\      &\hspace{5mm}g_t
  \leftarrow   \nabla_{\theta} f_t (\theta_{t-1})           \\
  &\hspace{5mm}if \: \lambda \neq 0
&\hspace{5mm}if \: \lambda \neq 0
      \      &\hspace{10mm} g_t \leftarrow g_t + \lambda
      \theta_{t-1}                            \      &\hspace{5mm}m_t
      \leftarrow   \beta_1 m_{t-1} + (1 - \beta_1) g_t
      \      &\hspace{5mm}u_t      \leftarrow   \mathrm{max}(\beta_2
      u_{t-1}, |g_{t}|+\epsilon)   \      &\hspace{5mm}\theta_t
      \leftarrow \theta_{t-1} - \frac{\gamma m_t}{(1-\beta^t_1) u_t}
      \      &\rule{110mm}{0.4pt}
      \[-1.ex]      &\bf{return} \:  \theta_t
      \[-1.ex]      &\rule{110mm}{0.4pt}
      \[-1.ex] \end{aligned}
For further details regarding the algorithm we refer to Adam: A
   Method for Stochastic Optimization.
Parameters:
      * params (iterable) -- iterable of parameters to optimize
        or dicts defining parameter groups
  * **lr** (*float**, **optional*) -- learning rate (default:
    2e-3)

  * **betas** (*Tuple**[**float**, **float**]**, **optional*) --
coefficients used for computing running averages of gradient
        and its square
  * **eps** (*float**, **optional*) -- term added to the
    denominator to improve numerical stability (default: 1e-8)

  * **weight_decay** (*float**, **optional*) -- weight decay (L2
    penalty) (default: 0)

  * **foreach** (*bool**, **optional*) -- whether foreach
    implementation of optimizer is used. If unspecified by the
    user (so foreach is None), we will try to use foreach over the
    for-loop implementation on CUDA, since it is usually
    significantly more performant. (default: None)

  * **maximize** (*bool**, **optional*) -- maximize the params
    based on the objective, instead of minimizing (default: False)

  * **differentiable** (*bool**, **optional*) -- whether autograd
    should occur through the optimizer step in training.
    Otherwise, the step() function runs in a torch.no_grad()
context. Setting to True can impair performance, so leave it
        False if you don't intend to run autograd through this
        instance (default: False)
add_param_group(param_group)
  Add a param group to the "Optimizer" s *param_groups*.

  This can be useful when fine tuning a pre-trained network as
  frozen layers can be made trainable and added to the "Optimizer"
  as training progresses.

  Parameters:
     **param_group** (*dict*) -- Specifies what Tensors should be
     optimized along with group specific optimization options.

load_state_dict(state_dict)
  Loads the optimizer state.

  Parameters:
     **state_dict** (*dict*) -- optimizer state. Should be an
     object returned from a call to "state_dict()".

register_step_post_hook(hook)
  Register an optimizer step post hook which will be called after
  optimizer step. It should have the following signature:

     hook(optimizer, args, kwargs) -> None
hook(optimizer, args, kwargs) -> None
  The "optimizer" argument is the optimizer instance being used.

  Parameters:
     **hook** (*Callable*) -- The user defined hook to be
     registered.

  Returns:
     a handle that can be used to remove the added hook by calling
     "handle.remove()"

  Return type:
     "torch.utils.hooks.RemoveableHandle"

register_step_pre_hook(hook)
  Register an optimizer step pre hook which will be called before
  optimizer step. It should have the following signature:

     hook(optimizer, args, kwargs) -> None or modified args and kwargs

  The "optimizer" argument is the optimizer instance being used.
  If args and kwargs are modified by the pre-hook, then the
  transformed values are returned as a tuple containing the
  new_args and new_kwargs.

  Parameters:
     **hook** (*Callable*) -- The user defined hook to be
     registered.

  Returns:
registered.
  Returns:
     a handle that can be used to remove the added hook by calling
     "handle.remove()"

  Return type:
     "torch.utils.hooks.RemoveableHandle"

state_dict()
  Returns the state of the optimizer as a "dict".

  It contains two entries:

  * state - a dict holding current optimization state. Its content
       differs between optimizer classes.

  * param_groups - a list containing all parameter groups where
    each
       parameter group is a dict

zero_grad(set_to_none=False)
  Sets the gradients of all optimized "torch.Tensor" s to zero.

  Parameters:
     **set_to_none** (*bool*) -- instead of setting to zero, set
     the grads to None. This will in general have lower memory
     footprint, and can modestly improve performance. However, it
     changes certain behaviors. For example: 1. When the user
     tries to access a gradient and perform manual ops on it, a
None attribute or a Tensor full of 0s will behave
         differently. 2. If the user requests
         "zero_grad(set_to_none=True)" followed by a backward pass,
         ".grad"s are guaranteed to be None for params that did not
         receive a gradient. 3. "torch.optim" optimizers have a
         different behavior if the gradient is 0 or None (in one case
         it does the step with a gradient of 0 and in the other it
         skips the step altogether).
ModuleDict
class torch.nn.ModuleDict(modules=None)
Holds submodules in a dictionary.
"ModuleDict" can be indexed like a regular Python dictionary, but
   modules it contains are properly registered, and will be visible by
   all "Module" methods.
"ModuleDict" is an ordered dictionary that respects


the order of insertion, and


in "update()", the order of the merged "OrderedDict", "dict"
     (started from Python 3.6) or another "ModuleDict" (the argument
     to "update()").


Note that "update()" with other unordered mapping types (e.g.,
   Python's plain "dict" before Python version 3.6) does not preserve
   the order of the merged mapping.
Parameters:
      modules (iterable, optional) -- a mapping (dictionary)
      of (string: module) or an iterable of key-value pairs of type
      (string, module)
Example:
  class MyModule(nn.Module):
      def __init__(self):
          super(MyModule, self).__init__()
super(MyModule, self).init()
              self.choices = nn.ModuleDict({
                      'conv': nn.Conv2d(10, 10, 3),
                      'pool': nn.MaxPool2d(3)
              })
              self.activations = nn.ModuleDict([
                      ['lrelu', nn.LeakyReLU()],
                      ['prelu', nn.PReLU()]
              ])
      def forward(self, x, choice, act):
          x = self.choices[choice](x)
          x = self.activations[act](x)
          return x

clear()
  Remove all items from the ModuleDict.

items()
  Return an iterable of the ModuleDict key/value pairs.

  Return type:
     *Iterable*[*Tuple*[str, *Module*]]

keys()
  Return an iterable of the ModuleDict keys.

  Return type:
     *Iterable*[str]

pop(key)
  Remove key from the ModuleDict and return its module.

  Parameters:
     **key** (*str*) -- key to pop from the ModuleDict

  Return type:
     *Module*
Return type:
         Module
update(modules)
  Update the "ModuleDict" with the key-value pairs from a mapping
  or an iterable, overwriting existing keys.

  Note:

    If "modules" is an "OrderedDict", a "ModuleDict", or an
    iterable of key-value pairs, the order of new elements in it
    is preserved.

  Parameters:
     **modules** (*iterable*) -- a mapping (dictionary) from
     string to "Module", or an iterable of key-value pairs of type
     (string, "Module")

values()
  Return an iterable of the ModuleDict values.

  Return type:
     *Iterable*[*Module*]
OneCycleLR
class torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr, total_steps=None, epochs=None, steps_per_epoch=None, pct_start=0.3, anneal_strategy='cos', cycle_momentum=True, base_momentum=0.85, max_momentum=0.95, div_factor=25.0, final_div_factor=10000.0, three_phase=False, last_epoch=- 1, verbose=False)
Sets the learning rate of each parameter group according to the
   1cycle learning rate policy. The 1cycle policy anneals the learning
   rate from an initial learning rate to some maximum learning rate
   and then from that maximum learning rate to some minimum learning
   rate much lower than the initial learning rate. This policy was
   initially described in the paper Super-Convergence: Very Fast
   Training of Neural Networks Using Large Learning Rates.
The 1cycle learning rate policy changes the learning rate after
   every batch. step should be called after a batch has been used
   for training.
This scheduler is not chainable.
This scheduler is not chainable.
Note also that the total number of steps in the cycle can be
   determined in one of two ways (listed in order of precedence):


A value for total_steps is explicitly provided.


A number of epochs (epochs) and a number of steps per epoch
      (steps_per_epoch) are provided. In this case, the number of
      total steps is inferred by total_steps = epochs *
      steps_per_epoch


You must either provide a value for total_steps or provide a value
   for both epochs and steps_per_epoch.
The default behaviour of this scheduler follows the fastai
   implementation of 1cycle, which claims that "unpublished work has
   shown even better results by using only two phases". To mimic the
   behaviour of the original paper instead, set "three_phase=True".
Parameters:
      * optimizer (Optimizer) -- Wrapped optimizer.
  * **max_lr** (*float** or **list*) -- Upper learning rate
    boundaries in the cycle for each parameter group.
total_steps (int) -- The total number of steps in the
        cycle. Note that if a value is not provided here, then it must
        be inferred by providing a value for epochs and
        steps_per_epoch. Default: None


epochs (int) -- The number of epochs to train for. This
    is used along with steps_per_epoch in order to infer the total
    number of steps in the cycle if a value for total_steps is not
    provided. Default: None


steps_per_epoch (int) -- The number of steps per epoch
    to train for. This is used along with epochs in order to infer
    the total number of steps in the cycle if a value for
    total_steps is not provided. Default: None


pct_start (float) -- The percentage of the cycle (in
    number of steps) spent increasing the learning rate. Default:
    0.3


anneal_strategy (str) -- {'cos', 'linear'} Specifies the
    annealing strategy: "cos" for cosine annealing, "linear" for
linear annealing. Default: 'cos'
  * **cycle_momentum** (*bool*) -- If "True", momentum is cycled
    inversely to learning rate between 'base_momentum' and
    'max_momentum'. Default: True

  * **base_momentum** (*float** or **list*) -- Lower momentum
    boundaries in the cycle for each parameter group. Note that
    momentum is cycled inversely to learning rate; at the peak of
    a cycle, momentum is 'base_momentum' and learning rate is
    'max_lr'. Default: 0.85

  * **max_momentum** (*float** or **list*) -- Upper momentum
    boundaries in the cycle for each parameter group.
    Functionally, it defines the cycle amplitude (max_momentum -
    base_momentum). Note that momentum is cycled inversely to
    learning rate; at the start of a cycle, momentum is
    'max_momentum' and learning rate is 'base_lr' Default: 0.95

  * **div_factor** (*float*) -- Determines the initial learning
rate via initial_lr = max_lr/div_factor Default: 25
  * **final_div_factor** (*float*) -- Determines the minimum
    learning rate via min_lr = initial_lr/final_div_factor
    Default: 1e4

  * **three_phase** (*bool*) -- If "True", use a third phase of
    the schedule to annihilate the learning rate according to
    'final_div_factor' instead of modifying the second phase (the
    first two phases will be symmetrical about the step indicated
    by 'pct_start').

  * **last_epoch** (*int*) -- The index of the last batch. This
    parameter is used when resuming a training job. Since *step()*
    should be invoked after each batch instead of after each
    epoch, this number represents the total number of *batches*
    computed, not the total number of epochs computed. When
    last_epoch=-1, the schedule is started from the beginning.
    Default: -1

  * **verbose** (*bool*) -- If "True", prints a message to stdout
for each update. Default: "False".
-[ Example ]-



data_loader = torch.utils.data.DataLoader(...)
optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=0.01, steps_per_epoch=len(data_loader), epochs=10)
for epoch in range(10):
    for batch in data_loader:
        train_batch(...)
        scheduler.step()



get_last_lr()
  Return last computed learning rate by current scheduler.

load_state_dict(state_dict)
  Loads the schedulers state.

  Parameters:
     **state_dict** (*dict*) -- scheduler state. Should be an
     object returned from a call to "state_dict()".

print_lr(is_verbose, group, lr, epoch=None)
  Display the current learning rate.

state_dict()
  Returns the state of the scheduler as a "dict".

  It contains an entry for every variable in self.__dict__ which
  is not the optimizer.
InstanceNorm3d
class torch.ao.nn.quantized.InstanceNorm3d(num_features, weight, bias, scale, zero_point, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False, device=None, dtype=None)
This is the quantized version of "InstanceNorm3d".
Additional args:
      * scale - quantization scale of the output, type: double.
  * **zero_point** - quantization zero point of the output, type:
    long.
torch.nn.functional.avg_pool1d
torch.nn.functional.avg_pool1d(input, kernel_size, stride=None, padding=0, ceil_mode=False, count_include_pad=True) -> Tensor
Applies a 1D average pooling over an input signal composed of
   several input planes.
See "AvgPool1d" for details and output shape.
Parameters:
      * input -- input tensor of shape (\text{minibatch} ,
        \text{in_channels} , iW)
  * **kernel_size** -- the size of the window. Can be a single
    number or a tuple *(kW,)*

  * **stride** -- the stride of the window. Can be a single number
    or a tuple *(sW,)*. Default: "kernel_size"

  * **padding** -- implicit zero paddings on both sides of the
    input. Can be a single number or a tuple *(padW,)*. Default: 0

  * **ceil_mode** -- when True, will use *ceil* instead of *floor*
    to compute the output shape. Default: "False"

  * **count_include_pad** -- when True, will include the zero-
padding in the averaging calculation. Default: "True"
Examples:
  >>> # pool of square window of size=3, stride=2
  >>> input = torch.tensor([[[1, 2, 3, 4, 5, 6, 7]]], dtype=torch.float32)
  >>> F.avg_pool1d(input, kernel_size=3, stride=2)
  tensor([[[ 2.,  4.,  6.]]])
torch.swapdims
torch.swapdims(input, dim0, dim1) -> Tensor
Alias for "torch.transpose()".
This function is equivalent to NumPy's swapaxes function.
Examples:
  >>> x = torch.tensor([[[0,1],[2,3]],[[4,5],[6,7]]])
  >>> x
  tensor([[[0, 1],
          [2, 3]],

          [[4, 5],
          [6, 7]]])
  >>> torch.swapdims(x, 0, 1)
  tensor([[[0, 1],
          [4, 5]],

          [[2, 3],
          [6, 7]]])
  >>> torch.swapdims(x, 0, 2)
  tensor([[[0, 4],
          [2, 6]],

          [[1, 5],
          [3, 7]]])