shrinath-suresh/blogs-docs-splitted huggingface dataset:
new function.



torch.dot                            # [D], [D] -> []
batched_dot = torch.func.vmap(torch.dot)  # [N, D], [N, D] -> [N]
x, y = torch.randn(2, 5), torch.randn(2, 5)
batched_dot(x, y)



"vmap()" can be helpful in hiding batch dimensions, leading to a
   simpler model authoring experience.



batch_size, feature_size = 3, 5
weights = torch.randn(feature_size, requires_grad=True)
def model(feature_vec):
    # Very simple linear model with activation
    return feature_vec.dot(weights).relu()
examples = torch.randn(batch_size, feature_size)
result = torch.vmap(model)(examples)



"vmap()" can also help vectorize computations that were previously
   difficult or impossible to batch. One example is higher-order
   gradient computation. The PyTorch autograd engine computes vjps
   (vector-Jacobian products). Computing a full Jacobian matrix for
   some function f: R^N -> R^N usually requires N calls to
"autograd.grad", one per Jacobian row. Using "vmap()", we can
   vectorize the whole computation, computing the Jacobian in a single
   call to "autograd.grad".



Setup
N = 5
f = lambda x: x ** 2
x = torch.randn(N, requires_grad=True)
y = f(x)
I_N = torch.eye(N)
Sequential approach
jacobian_rows = [torch.autograd.grad(y, x, v, retain_graph=True)[0]
                 for v in I_N.unbind()]
jacobian = torch.stack(jacobian_rows)
vectorized gradient computation
def get_vjp(v):
    return torch.autograd.grad(y, x, v)
jacobian = torch.vmap(get_vjp)(I_N)



"vmap()" can also be nested, producing an output with multiple
   batched dimensions



torch.dot                            # [D], [D] -> []
batched_dot = torch.vmap(torch.vmap(torch.dot))  # [N1, N0, D], [N1, N0, D] -> [N1, N0]
x, y = torch.randn(2, 3, 5), torch.randn(2, 3, 5)
batched_dot(x, y) # tensor of size [2, 3]
batched_dot(x, y) # tensor of size [2, 3]



If the inputs are not batched along the first dimension, "in_dims"
   specifies the dimension that each inputs are batched along as



torch.dot                            # [N], [N] -> []
batched_dot = torch.vmap(torch.dot, in_dims=1)  # [N, D], [N, D] -> [D]
x, y = torch.randn(2, 5), torch.randn(2, 5)
batched_dot(x, y)   # output is [5] instead of [2] if batched along the 0th dimension



If there are multiple inputs each of which is batched along
   different dimensions, "in_dims" must be a tuple with the batch
   dimension for each input as



torch.dot                            # [D], [D] -> []
batched_dot = torch.vmap(torch.dot, in_dims=(0, None))  # [N, D], [D] -> [N]
x, y = torch.randn(2, 5), torch.randn(5)
batched_dot(x, y) # second arg doesn't have a batch dim because in_dim[1] was None



If the input is a Python struct, "in_dims" must be a tuple
containing a struct matching the shape of the input:



f = lambda dict: torch.dot(dict['x'], dict['y'])
x, y = torch.randn(2, 5), torch.randn(5)
input = {'x': x, 'y': y}
batched_dot = torch.vmap(f, in_dims=({'x': 0, 'y': None},))
batched_dot(input)



By default, the output is batched along the first dimension.
   However, it can be batched along any dimension by using "out_dims"



f = lambda x: x ** 2
x = torch.randn(2, 5)
batched_pow = torch.vmap(f, out_dims=1)
batched_pow(x) # [5, 2]



For any function that uses kwargs, the returned function will not
   batch the kwargs but will accept kwargs



x = torch.randn([2, 5])
def fn(x, scale=4.):
  return x * scale
batched_pow = torch.vmap(fn)
assert torch.allclose(batched_pow(x), x * 4)
batched_pow(x, scale=x) # scale is not batched, output has shape [2, 2, 5]



Note:
 vmap does not provide general autobatching or handle variable-
torch.Tensor.bernoulli_
Tensor.bernoulli_(p=0.5, *, generator=None) -> Tensor
Fills each location of "self" with an independent sample from
   \text{Bernoulli}(\texttt{p}). "self" can have integral "dtype".
"p" should either be a scalar or tensor containing probabilities to
   be used for drawing the binary random number.
If it is a tensor, the \text{i}^{th} element of "self" tensor will
   be set to a value sampled from
   \text{Bernoulli}(\texttt{p_tensor[i]}). In this case p must have
   floating point "dtype".
See also "bernoulli()" and "torch.bernoulli()"
torch.logaddexp2
torch.logaddexp2(input, other, *, out=None) -> Tensor
Logarithm of the sum of exponentiations of the inputs in base-2.
Calculates pointwise \log_2\left(2^x + 2^y\right). See
   "torch.logaddexp()" for more details.
Parameters:
      * input (Tensor) -- the input tensor.
  * **other** (*Tensor*) -- the second input tensor

Keyword Arguments:
      out (Tensor, optional) -- the output tensor.
LazyInstanceNorm2d
class torch.nn.LazyInstanceNorm2d(eps=1e-05, momentum=0.1, affine=True, track_running_stats=True, device=None, dtype=None)
A "torch.nn.InstanceNorm2d" module with lazy initialization of the
   "num_features" argument of the "InstanceNorm2d" that is inferred
   from the "input.size(1)". The attributes that will be lazily
   initialized are weight, bias, running_mean and running_var.
Check the "torch.nn.modules.lazy.LazyModuleMixin" for further
   documentation on lazy modules and their limitations.
Parameters:
      * num_features -- C from an expected input of size (N, C, H,
        W) or (C, H, W)
  * **eps** (*float*) -- a value added to the denominator for
    numerical stability. Default: 1e-5

  * **momentum** (*float*) -- the value used for the running_mean
    and running_var computation. Default: 0.1

  * **affine** (*bool*) -- a boolean value that when set to
"True", this module has learnable affine parameters,
        initialized the same way as done for batch normalization.
        Default: "False".
  * **track_running_stats** (*bool*) -- a boolean value that when
    set to "True", this module tracks the running mean and
    variance, and when set to "False", this module does not track
    such statistics and always uses batch statistics in both
    training and eval modes. Default: "False"

Shape:
      * Input: (N, C, H, W) or (C, H, W)
  * Output: (N, C, H, W) or (C, H, W) (same shape as input)

cls_to_become
  alias of "InstanceNorm2d"
torch.isreal
torch.isreal(input) -> Tensor
Returns a new tensor with boolean elements representing if each
   element of "input" is real-valued or not. All real-valued types are
   considered real. Complex values are considered real when their
   imaginary part is 0.
Parameters:
      input (Tensor) -- the input tensor.
Returns:
      A boolean tensor that is True where "input" is real and False
      elsewhere
Example:
  >>> torch.isreal(torch.tensor([1, 1+1j, 2+0j]))
  tensor([True, False, True])
TransformerEncoderLayer
class torch.nn.TransformerEncoderLayer(d_model, nhead, dim_feedforward=2048, dropout=0.1, activation=, layer_norm_eps=1e-05, batch_first=False, norm_first=False, device=None, dtype=None)
TransformerEncoderLayer is made up of self-attn and feedforward
   network. This standard encoder layer is based on the paper
   "Attention Is All You Need". Ashish Vaswani, Noam Shazeer, Niki
   Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser,
   and Illia Polosukhin. 2017. Attention is all you need. In Advances
   in Neural Information Processing Systems, pages 6000-6010. Users
   may modify or implement in a different way during application.
Parameters:
      * d_model (int) -- the number of expected features in the
        input (required).
  * **nhead** (*int*) -- the number of heads in the
    multiheadattention models (required).

  * **dim_feedforward** (*int*) -- the dimension of the
feedforward network model (default=2048).
  * **dropout** (*float*) -- the dropout value (default=0.1).

  * **activation** (*Union**[**str**,
    **Callable**[**[**Tensor**]**, **Tensor**]**]*) -- the
    activation function of the intermediate layer, can be a string
    ("relu" or "gelu") or a unary callable. Default: relu

  * **layer_norm_eps** (*float*) -- the eps value in layer
    normalization components (default=1e-5).

  * **batch_first** (*bool*) -- If "True", then the input and
    output tensors are provided as (batch, seq, feature). Default:
    "False" (seq, batch, feature).

  * **norm_first** (*bool*) -- if "True", layer norm is done prior
    to attention and feedforward operations, respectively.
    Otherwise it's done after. Default: "False" (after).

Examples::
      >>> encoder_layer = nn.TransformerEncoderLayer(d_model=512, nhead=8)
      >>> src = torch.rand(10, 32, 512)
      >>> out = encoder_layer(src)
out = encoder_layer(src)



Alternatively, when "batch_first" is "True":
      >>> encoder_layer = nn.TransformerEncoderLayer(d_model=512, nhead=8, batch_first=True)
      >>> src = torch.rand(32, 10, 512)
      >>> out = encoder_layer(src)
Fast path:
      forward() will use a special optimized implementation if all of
      the following conditions are met:
  * Either autograd is disabled (using "torch.inference_mode" or
    "torch.no_grad") or no tensor argument "requires_grad"

  * training is disabled (using ".eval()")

  * batch_first is "True" and the input is batched (i.e.,
    "src.dim() == 3")

  * activation is one of: ""relu"", ""gelu"",
    "torch.functional.relu", or "torch.functional.gelu"

  * at most one of "src_mask" and "src_key_padding_mask" is passed

  * if src is a NestedTensor, neither "src_mask" nor
    "src_key_padding_mask" is passed

  * the two "LayerNorm" instances have a consistent "eps" value
(this will naturally be the case unless the caller has
        manually modified one without modifying the other)
  If the optimized implementation is in use, a NestedTensor can be
  passed for "src" to represent padding more efficiently than
  using a padding mask. In this case, a NestedTensor will be
  returned, and an additional speedup proportional to the fraction
  of the input that is padding can be expected.

forward(src, src_mask=None, src_key_padding_mask=None, is_causal=False)
  Pass the input through the encoder layer.

  Parameters:
     * **src** (*Tensor*) -- the sequence to the encoder layer
       (required).

     * **src_mask** (*Optional**[**Tensor**]*) -- the mask for the
       src sequence (optional).

     * **is_causal** (*bool*) -- If specified, applies a causal
       mask as src_mask. Mutually exclusive with providing
       src_mask. Default: "False".
MaxUnpool3d
class torch.nn.MaxUnpool3d(kernel_size, stride=None, padding=0)
Computes a partial inverse of "MaxPool3d".
"MaxPool3d" is not fully invertible, since the non-maximal values
   are lost. "MaxUnpool3d" takes in as input the output of "MaxPool3d"
   including the indices of the maximal values and computes a partial
   inverse in which all non-maximal values are set to zero.
Note:
 "MaxPool3d" can map several input sizes to the same output sizes.
 Hence, the inversion process can get ambiguous. To accommodate
 this, you can provide the needed output size as an additional
 argument "output_size" in the forward call. See the Inputs
 section below.

Parameters:
      * kernel_size (int or tuple) -- Size of the max
        pooling window.
  * **stride** (*int** or **tuple*) -- Stride of the max pooling
    window. It is set to "kernel_size" by default.

  * **padding** (*int** or **tuple*) -- Padding that was added to
the input
Inputs:
      * input: the input Tensor to invert
  * *indices*: the indices given out by "MaxPool3d"

  * *output_size* (optional): the targeted output size

Shape:
      * Input: (N, C, D_{in}, H_{in}, W_{in}) or (C, D_{in}, H_{in},
        W_{in}).
  * Output: (N, C, D_{out}, H_{out}, W_{out}) or (C, D_{out},
    H_{out}, W_{out}), where

       D_{out} = (D_{in} - 1) \times \text{stride[0]} - 2 \times
       \text{padding[0]} + \text{kernel\_size[0]}

       H_{out} = (H_{in} - 1) \times \text{stride[1]} - 2 \times
       \text{padding[1]} + \text{kernel\_size[1]}

       W_{out} = (W_{in} - 1) \times \text{stride[2]} - 2 \times
       \text{padding[2]} + \text{kernel\_size[2]}

    or as given by "output_size" in the call operator

Example:
  >>> # pool of square window of size=3, stride=2
  >>> pool = nn.MaxPool3d(3, stride=2, return_indices=True)
  >>> unpool = nn.MaxUnpool3d(3, stride=2)
torch.Tensor.is_leaf
Tensor.is_leaf
All Tensors that have "requires_grad" which is "False" will be leaf
   Tensors by convention.
For Tensors that have "requires_grad" which is "True", they will be
   leaf Tensors if they were created by the user. This means that they
   are not the result of an operation and so "grad_fn" is None.
Only leaf Tensors will have their "grad" populated during a call to
   "backward()". To get "grad" populated for non-leaf Tensors, you can
   use "retain_grad()".
Example:
  >>> a = torch.rand(10, requires_grad=True)
  >>> a.is_leaf
  True
  >>> b = torch.rand(10, requires_grad=True).cuda()
  >>> b.is_leaf
  False
  # b was created by the operation that cast a cpu Tensor into a cuda Tensor
  >>> c = torch.rand(10, requires_grad=True) + 2
  >>> c.is_leaf
  False
  # c was created by the addition operation
  >>> d = torch.rand(10).cuda()
  >>> d.is_leaf
  True
d.is_leaf
      True
      # d does not require gradients and so has no operation creating it (that is tracked by the autograd engine)
      >>> e = torch.rand(10).cuda().requires_grad_()
      >>> e.is_leaf
      True
      # e requires gradients and has no operations creating it
      >>> f = torch.rand(10, requires_grad=True, device="cuda")
      >>> f.is_leaf
      True
      # f requires grad, has no operation creating it
torch.jit.wait
torch.jit.wait(future)
Forces completion of a torch.jit.Future[T] asynchronous task,
   returning the result of the task. See "fork()" for docs and
   examples. :param future: an asynchronous task reference, created
   through torch.jit.fork :type future: torch.jit.Future[T]
Returns:
      the return value of the the completed task
Return type:
      T
torch.Tensor.reshape
Tensor.reshape(*shape) -> Tensor
Returns a tensor with the same data and number of elements as
   "self" but with the specified shape. This method returns a view if
   "shape" is compatible with the current shape. See
   "torch.Tensor.view()" on when it is possible to return a view.
See "torch.reshape()"
Parameters:
      shape (tuple of ints or int...) -- the desired shape
ObserverBase
class torch.quantization.observer.ObserverBase(dtype)
Base observer Module. Any observer implementation should derive
   from this class.
Concrete observers should follow the same API. In forward, they
   will update the statistics of the observed Tensor. And they should
   provide a calculate_qparams function that computes the
   quantization parameters given the collected statistics.
Parameters:
      dtype -- dtype argument to the quantize node needed to
      implement the reference model spec.
classmethod with_args(**kwargs)
  Wrapper that allows creation of class factories.

  This can be useful when there is a need to create classes with
  the same constructor arguments, but different instances. Can be
  used in conjunction with _callable_args

  Example:

     >>> Foo.with_args = classmethod(_with_args)
     >>> foo_builder = Foo.with_args(a=3, b=4).with_args(answer=42)
     >>> foo_instance1 = foo_builder()
foo_instance1 = foo_builder()
         >>> foo_instance2 = foo_builder()
         >>> id(foo_instance1) == id(foo_instance2)
         False



classmethod with_callable_args(**kwargs)
  Wrapper that allows creation of class factories args that need
  to be called at construction time.

  This can be useful when there is a need to create classes with
  the same constructor arguments, but different instances and
  those arguments should only be calculated at construction time.
  Can be used in conjunction with _with_args

  Example:

     >>> Foo.with_callable_args = classmethod(_with_callable_args)
     >>> Foo.with_args = classmethod(_with_args)
     >>> foo_builder = Foo.with_callable_args(cur_time=get_time_func).with_args(name="dan")
     >>> foo_instance1 = foo_builder()
     >>> # wait 50
     >>> foo_instance2 = foo_builder()
     >>> id(foo_instance1.creation_time) == id(foo_instance2.creation_time)
     False
torch.linalg.det
torch.linalg.det(A, *, out=None) -> Tensor
Computes the determinant of a square matrix.
Supports input of float, double, cfloat and cdouble dtypes. Also
   supports batches of matrices, and if "A" is a batch of matrices
   then the output has the same batch dimensions.
See also:
 "torch.linalg.slogdet()" computes the sign and natural logarithm
 of the absolute value of the determinant of square matrices.

Parameters:
      A (Tensor) -- tensor of shape (, n, n)* where *** is
      zero or more batch dimensions.
Keyword Arguments:
      out (Tensor, optional) -- output tensor. Ignored if
      None. Default: None.
Examples:
  >>> A = torch.randn(3, 3)
  >>> torch.linalg.det(A)
  tensor(0.0934)

  >>> A = torch.randn(3, 2, 2)
  >>> torch.linalg.det(A)
  tensor([1.1990, 0.4099, 0.7386])
TripletMarginWithDistanceLoss
class torch.nn.TripletMarginWithDistanceLoss(*, distance_function=None, margin=1.0, swap=False, reduction='mean')
Creates a criterion that measures the triplet loss given input
   tensors a, p, and n (representing anchor, positive, and negative
   examples, respectively), and a nonnegative, real-valued function
   ("distance function") used to compute the relationship between the
   anchor and positive example ("positive distance") and the anchor
   and negative example ("negative distance").
The unreduced loss (i.e., with "reduction" set to "'none'") can be
   described as:
  \ell(a, p, n) = L = \{l_1,\dots,l_N\}^\top, \quad l_i = \max
  \{d(a_i, p_i) - d(a_i, n_i) + {\rm margin}, 0\}

where N is the batch size; d is a nonnegative, real-valued function
   quantifying the closeness of two tensors, referred to as the
   "distance_function"; and margin is a nonnegative margin
representing the minimum difference between the positive and
   negative distances that is required for the loss to be 0.  The
   input tensors have N elements each and can be of any shape that the
   distance function can handle.
If "reduction" is not "'none'" (default "'mean'"), then:
  \ell(x, y) = \begin{cases}     \operatorname{mean}(L), &
  \text{if reduction} = \text{`mean';}\\
  \operatorname{sum}(L),  &  \text{if reduction} = \text{`sum'.}
  \end{cases}

See also "TripletMarginLoss", which computes the triplet loss for
   input tensors using the l_p distance as the distance function.
Parameters:
      * distance_function (Callable, optional) -- A
        nonnegative, real-valued function that quantifies the
        closeness of two tensors. If not specified,
        nn.PairwiseDistance will be used.  Default: "None"
  * **margin** (*float**, **optional*) -- A nonnegative margin
    representing the minimum difference between the positive and
negative distances required for the loss to be 0. Larger
        margins penalize cases where the negative examples are not
        distant enough from the anchors, relative to the positives.
        Default: 1.
  * **swap** (*bool**, **optional*) -- Whether to use the distance
    swap described in the paper *Learning shallow convolutional
    feature descriptors with triplet losses* by V. Balntas, E.
    Riba et al. If True, and if the positive example is closer to
    the negative example than the anchor is, swaps the positive
    example and the anchor in the loss computation. Default:
    "False".

  * **reduction** (*str**, **optional*) -- Specifies the
    (optional) reduction to apply to the output: "'none'" |
    "'mean'" | "'sum'". "'none'": no reduction will be applied,
    "'mean'": the sum of the output will be divided by the number
    of elements in the output, "'sum'": the output will be summed.
    Default: "'mean'"

Shape:
Default: "'mean'"
Shape:
      * Input: (N, *) where * represents any number of additional
        dimensions as supported by the distance function.
  * Output: A Tensor of shape (N) if "reduction" is "'none'", or a
    scalar otherwise.

Examples:
  >>> # Initialize embeddings
  >>> embedding = nn.Embedding(1000, 128)
  >>> anchor_ids = torch.randint(0, 1000, (1,))
  >>> positive_ids = torch.randint(0, 1000, (1,))
  >>> negative_ids = torch.randint(0, 1000, (1,))
  >>> anchor = embedding(anchor_ids)
  >>> positive = embedding(positive_ids)
  >>> negative = embedding(negative_ids)
  >>>
  >>> # Built-in Distance Function
  >>> triplet_loss = \
  >>>     nn.TripletMarginWithDistanceLoss(distance_function=nn.PairwiseDistance())
  >>> output = triplet_loss(anchor, positive, negative)
  >>> output.backward()
  >>>
  >>> # Custom Distance Function
  >>> def l_infinity(x1, x2):
def l_infinity(x1, x2):
      >>>     return torch.max(torch.abs(x1 - x2), dim=1).values
      >>>
      >>> triplet_loss = (
      >>>     nn.TripletMarginWithDistanceLoss(distance_function=l_infinity, margin=1.5))
      >>> output = triplet_loss(anchor, positive, negative)
      >>> output.backward()
      >>>
      >>> # Custom Distance Function (Lambda)
      >>> triplet_loss = (
      >>>     nn.TripletMarginWithDistanceLoss(
      >>>         distance_function=lambda x, y: 1.0 - F.cosine_similarity(x, y)))
      >>> output = triplet_loss(anchor, positive, negative)
      >>> output.backward()



Reference:
      V. Balntas, et al.: Learning shallow convolutional feature
      descriptors with triplet losses:
      http://www.bmva.org/bmvc/2016/papers/paper119/index.html
torch.rot90
torch.rot90(input, k=1, dims=[0, 1]) -> Tensor
Rotate an n-D tensor by 90 degrees in the plane specified by dims
   axis. Rotation direction is from the first towards the second axis
   if k > 0, and from the second towards the first for k < 0.
Parameters:
      * input (Tensor) -- the input tensor.
  * **k** (*int*) -- number of times to rotate. Default value is 1

  * **dims** (*a list** or **tuple*) -- axis to rotate. Default
    value is [0, 1]

Example:
  >>> x = torch.arange(4).view(2, 2)
  >>> x
  tensor([[0, 1],
          [2, 3]])
  >>> torch.rot90(x, 1, [0, 1])
  tensor([[1, 3],
          [0, 2]])

  >>> x = torch.arange(8).view(2, 2, 2)
  >>> x
  tensor([[[0, 1],
           [2, 3]],

          [[4, 5],
           [6, 7]]])
  >>> torch.rot90(x, 1, [1, 2])
  tensor([[[1, 3],
           [0, 2]],

          [[5, 7],
           [4, 6]]])
torch.nn.utils.prune.global_unstructured
torch.nn.utils.prune.global_unstructured(parameters, pruning_method, importance_scores=None, **kwargs)
Globally prunes tensors corresponding to all parameters in
   "parameters" by applying the specified "pruning_method". Modifies
   modules in place by:


adding a named buffer called "name+'_mask'" corresponding to the
      binary mask applied to the parameter "name" by the pruning
      method.


replacing the parameter "name" by its pruned version, while the
      original (unpruned) parameter is stored in a new parameter named
      "name+'_orig'".


Parameters:
      * parameters (Iterable of (module, name)
        tuples) -- parameters of the model to prune in a global
        fashion, i.e. by aggregating all weights prior to deciding
        which ones to prune. module must be of type "nn.Module", and
        name must be a string.
name must be a string.
  * **pruning_method** (*function*) -- a valid pruning function
    from this module, or a custom one implemented by the user that
    satisfies the implementation guidelines and has
    "PRUNING_TYPE='unstructured'".

  * **importance_scores** (*dict*) -- a dictionary mapping
    (module, name) tuples to the corresponding parameter's
    importance scores tensor. The tensor should be the same shape
    as the parameter, and is used for computing mask for pruning.
    If unspecified or None, the parameter will be used in place of
    its importance scores.

  * **kwargs** -- other keyword arguments such as: amount (int or
    float): quantity of parameters to prune across the specified
    parameters. If "float", should be between 0.0 and 1.0 and
    represent the fraction of parameters to prune. If "int", it
    represents the absolute number of parameters to prune.

Raises: