shrinath-suresh/blogs-docs-splitted huggingface dataset:
Transformer
class torch.nn.Transformer(d_model=512, nhead=8, num_encoder_layers=6, num_decoder_layers=6, dim_feedforward=2048, dropout=0.1, activation=, custom_encoder=None, custom_decoder=None, layer_norm_eps=1e-05, batch_first=False, norm_first=False, device=None, dtype=None)
A transformer model. User is able to modify the attributes as
   needed. The architecture is based on the paper "Attention Is All
   You Need". Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
   Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia
   Polosukhin. 2017. Attention is all you need. In Advances in Neural
   Information Processing Systems, pages 6000-6010.
Parameters:
      * d_model (int) -- the number of expected features in the
        encoder/decoder inputs (default=512).
  * **nhead** (*int*) -- the number of heads in the
    multiheadattention models (default=8).

  * **num_encoder_layers** (*int*) -- the number of sub-encoder-
layers in the encoder (default=6).
  * **num_decoder_layers** (*int*) -- the number of sub-decoder-
    layers in the decoder (default=6).

  * **dim_feedforward** (*int*) -- the dimension of the
    feedforward network model (default=2048).

  * **dropout** (*float*) -- the dropout value (default=0.1).

  * **activation** (*Union**[**str**,
    **Callable**[**[**Tensor**]**, **Tensor**]**]*) -- the
    activation function of encoder/decoder intermediate layer, can
    be a string ("relu" or "gelu") or a unary callable. Default:
    relu

  * **custom_encoder** (*Optional**[**Any**]*) -- custom encoder
    (default=None).

  * **custom_decoder** (*Optional**[**Any**]*) -- custom decoder
    (default=None).

  * **layer_norm_eps** (*float*) -- the eps value in layer
    normalization components (default=1e-5).

  * **batch_first** (*bool*) -- If "True", then the input and
output tensors are provided as (batch, seq, feature). Default:
        "False" (seq, batch, feature).
  * **norm_first** (*bool*) -- if "True", encoder and decoder
    layers will perform LayerNorms before other attention and
    feedforward operations, otherwise after. Default: "False"
    (after).

Examples::
      >>> transformer_model = nn.Transformer(nhead=16, num_encoder_layers=12)
      >>> src = torch.rand((10, 32, 512))
      >>> tgt = torch.rand((20, 32, 512))
      >>> out = transformer_model(src, tgt)
Note: A full example to apply nn.Transformer module for the word
   language model is available in
   https://github.com/pytorch/examples/tree/master/word_language_model
forward(src, tgt, src_mask=None, tgt_mask=None, memory_mask=None, src_key_padding_mask=None, tgt_key_padding_mask=None, memory_key_padding_mask=None)
  Take in and process masked source/target sequences.

  Parameters:
     * **src** (*Tensor*) -- the sequence to the encoder
(required).
     * **tgt** (*Tensor*) -- the sequence to the decoder
       (required).

     * **src_mask** (*Optional**[**Tensor**]*) -- the additive
       mask for the src sequence (optional).

     * **tgt_mask** (*Optional**[**Tensor**]*) -- the additive
       mask for the tgt sequence (optional).

     * **memory_mask** (*Optional**[**Tensor**]*) -- the additive
       mask for the encoder output (optional).

     * **src_key_padding_mask** (*Optional**[**Tensor**]*) -- the
       ByteTensor mask for src keys per batch (optional).

     * **tgt_key_padding_mask** (*Optional**[**Tensor**]*) -- the
       ByteTensor mask for tgt keys per batch (optional).

     * **memory_key_padding_mask** (*Optional**[**Tensor**]*) --
       the ByteTensor mask for memory keys per batch (optional).

  Return type:
     *Tensor*

  Shape:
     * src: (S, E) for unbatched input, (S, N, E) if
batch_first=False or (N, S, E) if batch_first=True.
     * tgt: (T, E) for unbatched input, (T, N, E) if
       *batch_first=False* or *(N, T, E)* if *batch_first=True*.

     * src_mask: (S, S) or (N\cdot\text{num\_heads}, S, S).

     * tgt_mask: (T, T) or (N\cdot\text{num\_heads}, T, T).

     * memory_mask: (T, S).

     * src_key_padding_mask: (S) for unbatched input otherwise (N,
       S).

     * tgt_key_padding_mask: (T) for unbatched input otherwise (N,
       T).

     * memory_key_padding_mask: (S) for unbatched input otherwise
       (N, S).

     Note: [src/tgt/memory]_mask ensures that position i is
     allowed to attend the unmasked positions. If a ByteTensor is
     provided, the non-zero positions are not allowed to attend
     while the zero positions will be unchanged. If a BoolTensor
     is provided, positions with "True" are not allowed to attend
while "False" values will be unchanged. If a FloatTensor is
         provided, it will be added to the attention weight.
         [src/tgt/memory]_key_padding_mask provides specified elements
         in the key to be ignored by the attention. If a ByteTensor is
         provided, the non-zero positions will be ignored while the
         zero positions will be unchanged. If a BoolTensor is
         provided, the positions with the value of "True" will be
         ignored while the position with the value of "False" will be
         unchanged.
     * output: (T, E) for unbatched input, (T, N, E) if
       *batch_first=False* or *(N, T, E)* if *batch_first=True*.

     Note: Due to the multi-head attention architecture in the
     transformer model, the output sequence length of a
     transformer is same as the input sequence (i.e. target)
     length of the decoder.

     where S is the source sequence length, T is the target
sequence length, N is the batch size, E is the feature number
  -[ Examples ]-

  >>> output = transformer_model(src, tgt, src_mask=src_mask, tgt_mask=tgt_mask)

static generate_square_subsequent_mask(sz, device='cpu')
  Generate a square mask for the sequence. The masked positions
  are filled with float('-inf'). Unmasked positions are filled
  with float(0.0).

  Return type:
     *Tensor*
torch.sparse_csr_tensor
torch.sparse_csr_tensor(crow_indices, col_indices, values, size=None, *, dtype=None, device=None, requires_grad=False, check_invariants=None) -> Tensor
Constructs a sparse tensor in CSR (Compressed Sparse Row) with
   specified values at the given "crow_indices" and "col_indices".
   Sparse matrix multiplication operations in CSR format are typically
   faster than that for sparse tensors in COO format. Make you have a
   look at the note on the data type of the indices.
Note:
 If the "device" argument is not specified the device of the given
 "values" and indices tensor(s) must match. If, however, the
 argument is specified the input Tensors will be converted to the
 given device and in turn determine the device of the constructed
 sparse tensor.

Parameters:
      * crow_indices (array_like) -- (B+1)-dimensional array of
        size "(*batchsize, nrows + 1)".  The last element of each
batch is the number of non-zeros. This tensor encodes the
        index in values and col_indices depending on where the given
        row starts. Each successive number in the tensor subtracted by
        the number before it denotes the number of elements in a given
        row.
  * **col_indices** (*array_like*) -- Column co-ordinates of each
    element in values. (B+1)-dimensional tensor with the same
    length as values.

  * **values** (*array_list*) -- Initial values for the tensor.
    Can be a list, tuple, NumPy "ndarray", scalar, and other types
    that represents a (1+K)-dimensional tensor where "K" is the
    number of dense dimensions.

  * **size** (list, tuple, "torch.Size", optional) -- Size of the
    sparse tensor: "(*batchsize, nrows, ncols, *densesize)". If
    not provided, the size will be inferred as the minimum size
    big enough to hold all non-zero elements.

Keyword Arguments:
Keyword Arguments:
      * dtype ("torch.dtype", optional) -- the desired data type
        of returned tensor.  Default: if None, infers data type from
        "values".
  * **device** ("torch.device", optional) -- the desired device of
    returned tensor.  Default: if None, uses the current device
    for the default tensor type (see
    "torch.set_default_tensor_type()"). "device" will be the CPU
    for CPU tensor types and the current CUDA device for CUDA
    tensor types.

  * **requires_grad** (*bool**, **optional*) -- If autograd should
    record operations on the returned tensor. Default: "False".

  * **check_invariants** (*bool**, **optional*) -- If sparse
    tensor invariants are checked. Default: as returned by
    "torch.sparse.check_sparse_tensor_invariants.is_enabled()",
    initially False.

Example::
      >>> crow_indices = [0, 2, 4]
      >>> col_indices = [0, 1, 0, 1]
      >>> values = [1, 2, 3, 4]
values = [1, 2, 3, 4]
      >>> torch.sparse_csr_tensor(torch.tensor(crow_indices, dtype=torch.int64),
      ...                         torch.tensor(col_indices, dtype=torch.int64),
      ...                         torch.tensor(values), dtype=torch.double)
      tensor(crow_indices=tensor([0, 2, 4]),
             col_indices=tensor([0, 1, 0, 1]),
             values=tensor([1., 2., 3., 4.]), size=(2, 2), nnz=4,
             dtype=torch.float64, layout=torch.sparse_csr)
torch.column_stack
torch.column_stack(tensors, *, out=None) -> Tensor
Creates a new tensor by horizontally stacking the tensors in
   "tensors".
Equivalent to "torch.hstack(tensors)", except each zero or one
   dimensional tensor "t" in "tensors" is first reshaped into a
   "(t.numel(), 1)" column before being stacked horizontally.
Parameters:
      tensors (sequence of Tensors) -- sequence of tensors to
      concatenate
Keyword Arguments:
      out (Tensor, optional) -- the output tensor.
Example:
  >>> a = torch.tensor([1, 2, 3])
  >>> b = torch.tensor([4, 5, 6])
  >>> torch.column_stack((a, b))
  tensor([[1, 4],
      [2, 5],
      [3, 6]])
  >>> a = torch.arange(5)
  >>> b = torch.arange(10).reshape(5, 2)
  >>> torch.column_stack((a, b, b))
  tensor([[0, 0, 1, 0, 1],
          [1, 2, 3, 2, 3],
          [2, 4, 5, 4, 5],
          [3, 6, 7, 6, 7],
          [4, 8, 9, 8, 9]])
torch.nn.utils.parametrizations.orthogonal
torch.nn.utils.parametrizations.orthogonal(module, name='weight', orthogonal_map=None, *, use_trivialization=True)
Applies an orthogonal or unitary parametrization to a matrix or a
   batch of matrices.
Letting \mathbb{K} be \mathbb{R} or \mathbb{C}, the parametrized
   matrix Q \in \mathbb{K}^{m \times n} is orthogonal as
  \begin{align*}     Q^{\text{H}}Q &= \mathrm{I}_n
  \mathrlap{\qquad \text{if }m \geq n}\\     QQ^{\text{H}} &=
  \mathrm{I}_m \mathrlap{\qquad \text{if }m < n} \end{align*}

where Q^{\text{H}} is the conjugate transpose when Q is complex and
   the transpose when Q is real-valued, and \mathrm{I}_n is the
   n-dimensional identity matrix. In plain words, Q will have
   orthonormal columns whenever m \geq n and orthonormal rows
   otherwise.
If the tensor has more than two dimensions, we consider it as a
   batch of matrices of shape (..., m, n).
batch of matrices of shape (..., m, n).
The matrix Q may be parametrized via three different
   "orthogonal_map" in terms of the original tensor:


""matrix_exp""/""cayley"": the "matrix_exp()" Q = \exp(A) and the
     Cayley map Q = (\mathrm{I}_n + A/2)(\mathrm{I}_n - A/2)^{-1} are
     applied to a skew-symmetric A to give an orthogonal matrix.


""householder"": computes a product of Householder reflectors
     ("householder_product()").


""matrix_exp""/""cayley"" often make the parametrized weight
   converge faster than ""householder"", but they are slower to
   compute for very thin or very wide matrices.
If "use_trivialization=True" (default), the parametrization
   implements the "Dynamic Trivialization Framework", where an extra
   matrix B \in \mathbb{K}^{n \times n} is stored under
   "module.parametrizations.weight[0].base". This helps the
   convergence of the parametrized layer at the expense of some extra
memory use. See Trivializations for Gradient-Based Optimization on
   Manifolds .
Initial value of Q: If the original tensor is not parametrized and
   "use_trivialization=True" (default), the initial value of Q is that
   of the original tensor if it is orthogonal (or unitary in the
   complex case) and it is orthogonalized via the QR decomposition
   otherwise (see "torch.linalg.qr()"). Same happens when it is not
   parametrized and "orthogonal_map="householder"" even when
   "use_trivialization=False". Otherwise, the initial value is the
   result of the composition of all the registered parametrizations
   applied to the original tensor.
Note:
 This function is implemented using the parametrization
 functionality in "register_parametrization()".

Parameters:
      * module (nn.Module) -- module on which to register the
        parametrization.
  * **name** (*str**, **optional*) -- name of the tensor to make
    orthogonal. Default: ""weight"".
orthogonal. Default: ""weight"".
  * **orthogonal_map** (*str**, **optional*) -- One of the
    following: ""matrix_exp"", ""cayley"", ""householder"".
    Default: ""matrix_exp"" if the matrix is square or complex,
    ""householder"" otherwise.

  * **use_trivialization** (*bool**, **optional*) -- whether to
    use the dynamic trivialization framework. Default: "True".

Returns:
      The original module with an orthogonal parametrization
      registered to the specified weight
Return type:
      Module
Example:
  >>> orth_linear = orthogonal(nn.Linear(20, 40))
  >>> orth_linear
  ParametrizedLinear(
  in_features=20, out_features=40, bias=True
  (parametrizations): ModuleDict(
      (weight): ParametrizationList(
      (0): _Orthogonal()
      )
  )
  )
  >>> Q = orth_linear.weight
  >>> torch.dist(Q.T @ Q, torch.eye(20))
  tensor(4.9332e-07)
SyncBatchNorm
class torch.nn.SyncBatchNorm(num_features, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True, process_group=None, device=None, dtype=None)
Applies Batch Normalization over a N-Dimensional input (a mini-
   batch of [N-2]D inputs with additional channel dimension) as
   described in the paper Batch Normalization: Accelerating Deep
   Network Training by Reducing Internal Covariate Shift .
  y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}}
  * \gamma + \beta

The mean and standard-deviation are calculated per-dimension over
   all mini-batches of the same process groups. \gamma and \beta are
   learnable parameter vectors of size C (where C is the input
   size). By default, the elements of \gamma are sampled from
   \mathcal{U}(0, 1) and the elements of \beta are set to 0. The
   standard-deviation is calculated via the biased estimator,
   equivalent to torch.var(input, unbiased=False).
Also by default, during training this layer keeps running estimates
   of its computed mean and variance, which are then used for
   normalization during evaluation. The running estimates are kept
   with a default "momentum" of 0.1.
If "track_running_stats" is set to "False", this layer then does
   not keep running estimates, and batch statistics are instead used
   during evaluation time as well.
Note:
 This "momentum" argument is different from one used in optimizer
 classes and the conventional notion of momentum. Mathematically,
 the update rule for running statistics here is \hat{x}_\text{new}
 = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times
 x_t, where \hat{x} is the estimated statistic and x_t is the new
 observed value.

Because the Batch Normalization is done for each channel in the "C"
   dimension, computing statistics on "(N, +)" slices, it's common
   terminology to call this Volumetric Batch Normalization or Spatio-
temporal Batch Normalization.
Currently "SyncBatchNorm" only supports "DistributedDataParallel"
   (DDP) with single GPU per process. Use
   "torch.nn.SyncBatchNorm.convert_sync_batchnorm()" to convert
   "BatchNorm*D" layer to "SyncBatchNorm" before wrapping Network with
   DDP.
Parameters:
      * num_features (int) -- C from an expected input of size
        (N, C, +)
  * **eps** (*float*) -- a value added to the denominator for
    numerical stability. Default: "1e-5"

  * **momentum** (*float*) -- the value used for the running_mean
    and running_var computation. Can be set to "None" for
    cumulative moving average (i.e. simple average). Default: 0.1

  * **affine** (*bool*) -- a boolean value that when set to
    "True", this module has learnable affine parameters. Default:
    "True"

  * **track_running_stats** (*bool*) -- a boolean value that when
    set to "True", this module tracks the running mean and
variance, and when set to "False", this module does not track
        such statistics, and initializes statistics buffers
        "running_mean" and "running_var" as "None". When these buffers
        are "None", this module always uses batch statistics. in both
        training and eval modes. Default: "True"
  * **process_group** (*Optional**[**Any**]*) -- synchronization
    of stats happen within each process group individually.
    Default behavior is synchronization across the whole world

Shape:
      * Input: (N, C, +)
  * Output: (N, C, +) (same shape as input)

Note:
 Synchronization of batchnorm statistics occurs only while
 training, i.e. synchronization is disabled when "model.eval()" is
 set or if "self.training" is otherwise "False".

Examples:
  >>> # With Learnable Parameters
  >>> m = nn.SyncBatchNorm(100)
  >>> # creating process group (optional)
  >>> # ranks is a list of int identifying rank ids.
ranks = list(range(8))
      >>> r1, r2 = ranks[:4], ranks[4:]
      >>> # Note: every rank calls into new_group for every
      >>> # process group created, even if that rank is not
      >>> # part of the group.
      >>> process_groups = [torch.distributed.new_group(pids) for pids in [r1, r2]]
      >>> process_group = process_groups[0 if dist.get_rank() <= 3 else 1]
      >>> # Without Learnable Parameters
      >>> m = nn.BatchNorm3d(100, affine=False, process_group=process_group)
      >>> input = torch.randn(20, 100, 35, 45, 10)
      >>> output = m(input)



  >>> # network is nn.BatchNorm layer
  >>> sync_bn_network = nn.SyncBatchNorm.convert_sync_batchnorm(network, process_group)
  >>> # only single gpu per process is currently supported
  >>> ddp_sync_bn_network = torch.nn.parallel.DistributedDataParallel(
  >>>                         sync_bn_network,
  >>>                         device_ids=[args.local_rank],
output_device=args.local_rank)




classmethod convert_sync_batchnorm(module, process_group=None)
  Helper function to convert all "BatchNorm*D" layers in the model
  to "torch.nn.SyncBatchNorm" layers.

  Parameters:
     * **module** (*nn.Module*) -- module containing one or more
       "BatchNorm*D" layers

     * **process_group** (*optional*) -- process group to scope
       synchronization, default is the whole world

  Returns:
     The original "module" with the converted
     "torch.nn.SyncBatchNorm" layers. If the original "module" is
     a "BatchNorm*D" layer, a new "torch.nn.SyncBatchNorm" layer
     object will be returned instead.

  Example:

     >>> # Network with nn.BatchNorm layer
     >>> module = torch.nn.Sequential(
     >>>            torch.nn.Linear(20, 100),
     >>>            torch.nn.BatchNorm1d(100),
     >>>          ).cuda()
     >>> # creating process group (optional)
creating process group (optional)
     >>> # ranks is a list of int identifying rank ids.
     >>> ranks = list(range(8))
     >>> r1, r2 = ranks[:4], ranks[4:]
     >>> # Note: every rank calls into new_group for every
     >>> # process group created, even if that rank is not
     >>> # part of the group.
     >>> process_groups = [torch.distributed.new_group(pids) for pids in [r1, r2]]
     >>> process_group = process_groups[0 if dist.get_rank() <= 3 else 1]
     >>> sync_bn_module = torch.nn.SyncBatchNorm.convert_sync_batchnorm(module, process_group)
LambdaLR
class torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda, last_epoch=- 1, verbose=False)
Sets the learning rate of each parameter group to the initial lr
   times a given function. When last_epoch=-1, sets initial lr as lr.
Parameters:
      * optimizer (Optimizer) -- Wrapped optimizer.
  * **lr_lambda** (*function** or **list*) -- A function which
    computes a multiplicative factor given an integer parameter
    epoch, or a list of such functions, one for each group in
    optimizer.param_groups.

  * **last_epoch** (*int*) -- The index of last epoch. Default:
    -1.

  * **verbose** (*bool*) -- If "True", prints a message to stdout
    for each update. Default: "False".

-[ Example ]-



Assuming optimizer has two groups.
lambda1 = lambda epoch: epoch // 30
lambda2 = lambda epoch: 0.95 ** epoch
scheduler = LambdaLR(optimizer, lr_lambda=[lambda1, lambda2])
for epoch in range(100):
for epoch in range(100):
    train(...)
    validate(...)
    scheduler.step()



get_last_lr()
  Return last computed learning rate by current scheduler.

load_state_dict(state_dict)
  Loads the schedulers state.

  When saving or loading the scheduler, please make sure to also
  save or load the state of the optimizer.

  Parameters:
     **state_dict** (*dict*) -- scheduler state. Should be an
     object returned from a call to "state_dict()".

print_lr(is_verbose, group, lr, epoch=None)
  Display the current learning rate.

state_dict()
  Returns the state of the scheduler as a "dict".

  It contains an entry for every variable in self.__dict__ which
  is not the optimizer. The learning rate lambda functions will
  only be saved if they are callable objects and not if they are
  functions or lambdas.

  When saving or loading the scheduler, please make sure to also
torch.eye
torch.eye(n, m=None, *, out=None, dtype=None, layout=torch.strided, device=None, requires_grad=False) -> Tensor
Returns a 2-D tensor with ones on the diagonal and zeros elsewhere.
Parameters:
      * n (int) -- the number of rows
  * **m** (*int**, **optional*) -- the number of columns with
    default being "n"

Keyword Arguments:
      * out (Tensor, optional) -- the output tensor.
  * **dtype** ("torch.dtype", optional) -- the desired data type
    of returned tensor. Default: if "None", uses a global default
    (see "torch.set_default_tensor_type()").

  * **layout** ("torch.layout", optional) -- the desired layout of
    returned Tensor. Default: "torch.strided".

  * **device** ("torch.device", optional) -- the desired device of
    returned tensor. Default: if "None", uses the current device
    for the default tensor type (see
    "torch.set_default_tensor_type()"). "device" will be the CPU
for CPU tensor types and the current CUDA device for CUDA
        tensor types.
  * **requires_grad** (*bool**, **optional*) -- If autograd should
    record operations on the returned tensor. Default: "False".

Returns:
      A 2-D tensor with ones on the diagonal and zeros elsewhere
Return type:
      Tensor
Example:
  >>> torch.eye(3)
  tensor([[ 1.,  0.,  0.],
          [ 0.,  1.,  0.],
          [ 0.,  0.,  1.]])
Adagrad
class torch.optim.Adagrad(params, lr=0.01, lr_decay=0, weight_decay=0, initial_accumulator_value=0, eps=1e-10, foreach=None, *, maximize=False, differentiable=False)
Implements Adagrad algorithm.
  \begin{aligned}      &\rule{110mm}{0.4pt}
  \\      &\textbf{input}      : \gamma \text{ (lr)}, \: \theta_0
  \text{ (params)}, \: f(\theta)          \text{ (objective)}, \:
  \lambda \text{ (weight decay)},                          \\
  &\hspace{12mm}    \tau \text{ (initial accumulator value)}, \:
  \eta\text{ (lr decay)}\\      &\textbf{initialize} :
  state\_sum_0 \leftarrow 0                             \\[-1.ex]
  &\rule{110mm}{0.4pt}
  \\      &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \:
  \textbf{do}                         \\      &\hspace{5mm}g_t
  \leftarrow   \nabla_{\theta} f_t (\theta_{t-1})           \\
  &\hspace{5mm} \tilde{\gamma}    \leftarrow \gamma / (1 +(t-1)