shrinath-suresh/blogs-docs-splitted huggingface dataset:
training is disabled (using ".eval()")


"add_bias_kv" is "False"


"add_zero_attn" is "False"


"batch_first" is "True" and the input is batched


"kdim" and "vdim" are equal to "embed_dim"


if a NestedTensor is passed, neither "key_padding_mask" nor
     "attn_mask" is passed


autocast is disabled


If the optimized implementation is in use, a NestedTensor can be
   passed for "query"/"key"/"value" to represent padding more
   efficiently than using a padding mask. In this case, a NestedTensor
   will be returned, and an additional speedup proportional to the
   fraction of the input that is padding can be expected.
Parameters:
      * embed_dim -- Total dimension of the model.
  * **num_heads** -- Number of parallel attention heads. Note that
    "embed_dim" will be split across "num_heads" (i.e. each head
    will have dimension "embed_dim // num_heads").

  * **dropout** -- Dropout probability on "attn_output_weights".
Default: "0.0" (no dropout).
  * **bias** -- If specified, adds bias to input / output
    projection layers. Default: "True".

  * **add_bias_kv** -- If specified, adds bias to the key and
    value sequences at dim=0. Default: "False".

  * **add_zero_attn** -- If specified, adds a new batch of zeros
    to the key and value sequences at dim=1. Default: "False".

  * **kdim** -- Total number of features for keys. Default: "None"
    (uses "kdim=embed_dim").

  * **vdim** -- Total number of features for values. Default:
    "None" (uses "vdim=embed_dim").

  * **batch_first** -- If "True", then the input and output
    tensors are provided as (batch, seq, feature). Default:
    "False" (seq, batch, feature).

Examples:
  >>> multihead_attn = nn.MultiheadAttention(embed_dim, num_heads)
  >>> attn_output, attn_output_weights = multihead_attn(query, key, value)
forward(query, key, value, key_padding_mask=None, need_weights=True, attn_mask=None, average_attn_weights=True, is_causal=False)
  Parameters:
     * **query** (*Tensor*) -- Query embeddings of shape (L, E_q)
       for unbatched input, (L, N, E_q) when "batch_first=False"
       or (N, L, E_q) when "batch_first=True", where L is the
       target sequence length, N is the batch size, and E_q is the
       query embedding dimension "embed_dim". Queries are compared
       against key-value pairs to produce the output. See
       "Attention Is All You Need" for more details.

     * **key** (*Tensor*) -- Key embeddings of shape (S, E_k) for
       unbatched input, (S, N, E_k) when "batch_first=False" or
       (N, S, E_k) when "batch_first=True", where S is the source
       sequence length, N is the batch size, and E_k is the key
       embedding dimension "kdim". See "Attention Is All You Need"
       for more details.
for more details.
     * **value** (*Tensor*) -- Value embeddings of shape (S, E_v)
       for unbatched input, (S, N, E_v) when "batch_first=False"
       or (N, S, E_v) when "batch_first=True", where S is the
       source sequence length, N is the batch size, and E_v is the
       value embedding dimension "vdim". See "Attention Is All You
       Need" for more details.

     * **key_padding_mask** (*Optional**[**Tensor**]*) -- If
       specified, a mask of shape (N, S) indicating which elements
       within "key" to ignore for the purpose of attention (i.e.
       treat as "padding"). For unbatched *query*, shape should be
       (S). Binary and byte masks are supported. For a binary
       mask, a "True" value indicates that the corresponding "key"
       value will be ignored for the purpose of attention. For a
       float mask, it will be directly added to the corresponding
       "key" value.
"key" value.
     * **need_weights** (*bool*) -- If specified, returns
       "attn_output_weights" in addition to "attn_outputs".
       Default: "True".

     * **attn_mask** (*Optional**[**Tensor**]*) -- If specified, a
       2D or 3D mask preventing attention to certain positions.
       Must be of shape (L, S) or (N\cdot\text{num\_heads}, L, S),
       where N is the batch size, L is the target sequence length,
       and S is the source sequence length. A 2D mask will be
       broadcasted across the batch while a 3D mask allows for a
       different mask for each entry in the batch. Binary, byte,
       and float masks are supported. For a binary mask, a "True"
       value indicates that the corresponding position is not
       allowed to attend. For a byte mask, a non-zero value
       indicates that the corresponding position is not allowed to
       attend. For a float mask, the mask values will be added to
the attention weight.
     * **is_causal** (*bool*) -- If specified, applies a causal
       mask as attention mask. Mutually exclusive with providing
       attn_mask. Default: "False".

     * **average_attn_weights** (*bool*) -- If true, indicates
       that the returned "attn_weights" should be averaged across
       heads. Otherwise, "attn_weights" are provided separately
       per head. Note that this flag only has an effect when
       "need_weights=True". Default: "True" (i.e. average weights
       across heads)

  Return type:
     *Tuple*[*Tensor*, *Optional*[*Tensor*]]

  Outputs:
     * **attn_output** - Attention outputs of shape (L, E) when
       input is unbatched, (L, N, E) when "batch_first=False" or
       (N, L, E) when "batch_first=True", where L is the target
       sequence length, N is the batch size, and E is the
       embedding dimension "embed_dim".
embedding dimension "embed_dim".
     * **attn_output_weights** - Only returned when
       "need_weights=True". If "average_attn_weights=True",
       returns attention weights averaged across heads of shape
       (L, S) when input is unbatched or (N, L, S), where N is the
       batch size, L is the target sequence length, and S is the
       source sequence length. If "average_attn_weights=False",
       returns attention weights per head of shape
       (\text{num\_heads}, L, S) when input is unbatched or (N,
       \text{num\_heads}, L, S).

     Note:

       *batch_first* argument is ignored for unbatched inputs.

merge_masks(attn_mask, key_padding_mask, query)
  Determine mask type and combine masks if necessary. If only one
  mask is provided, that mask and the corresponding mask type will
  be returned. If both masks are provided, they will be both
  expanded to shape "(batch_size, num_heads, seq_len, seq_len)",
combined with logical "or" and mask type 2 will be returned
      :param attn_mask: attention mask of shape "(seq_len, seq_len)",
      mask type 0 :param key_padding_mask: padding mask of shape
      "(batch_size, seq_len)", mask type 1 :param query: query
      embeddings of shape "(batch_size, seq_len, embed_dim)"
  Returns:
     merged mask mask_type: merged mask type (0, 1, or 2)

  Return type:
     merged_mask
torch.bitwise_xor
torch.bitwise_xor(input, other, *, out=None) -> Tensor
Computes the bitwise XOR of "input" and "other". The input tensor
   must be of integral or Boolean types. For bool tensors, it computes
   the logical XOR.
Parameters:
      * input -- the first input tensor
  * **other** -- the second input tensor

Keyword Arguments:
      out (Tensor, optional) -- the output tensor.
Example:
  >>> torch.bitwise_xor(torch.tensor([-1, -2, 3], dtype=torch.int8), torch.tensor([1, 0, 3], dtype=torch.int8))
  tensor([-2, -2,  0], dtype=torch.int8)
  >>> torch.bitwise_xor(torch.tensor([True, True, False]), torch.tensor([False, True, False]))
  tensor([ True, False, False])
torch.cuda.list_gpu_processes
torch.cuda.list_gpu_processes(device=None)
Returns a human-readable printout of the running processes and
   their GPU memory use for a given device.
This can be useful to display periodically during training, or when
   handling out-of-memory exceptions.
Parameters:
      device (torch.device or int, optional) -- selected
      device. Returns printout for the current device, given by
      "current_device()", if "device" is "None" (default).
Return type:
      str
torch.full_like
torch.full_like(input, fill_value, *, dtype=None, layout=torch.strided, device=None, requires_grad=False, memory_format=torch.preserve_format) -> Tensor
Returns a tensor with the same size as "input" filled with
   "fill_value". "torch.full_like(input, fill_value)" is equivalent to
   "torch.full(input.size(), fill_value, dtype=input.dtype,
   layout=input.layout, device=input.device)".
Parameters:
      * input (Tensor) -- the size of "input" will determine
        size of the output tensor.
  * **fill_value** -- the number to fill the output tensor with.

Keyword Arguments:
      * dtype ("torch.dtype", optional) -- the desired data type
        of returned Tensor. Default: if "None", defaults to the dtype
        of "input".
  * **layout** ("torch.layout", optional) -- the desired layout of
    returned tensor. Default: if "None", defaults to the layout of
    "input".
"input".
  * **device** ("torch.device", optional) -- the desired device of
    returned tensor. Default: if "None", defaults to the device of
    "input".

  * **requires_grad** (*bool**, **optional*) -- If autograd should
    record operations on the returned tensor. Default: "False".

  * **memory_format** ("torch.memory_format", optional) -- the
    desired memory format of returned Tensor. Default:
    "torch.preserve_format".
ConvTranspose2d
class torch.nn.ConvTranspose2d(in_channels, out_channels, kernel_size, stride=1, padding=0, output_padding=0, groups=1, bias=True, dilation=1, padding_mode='zeros', device=None, dtype=None)
Applies a 2D transposed convolution operator over an input image
   composed of several input planes.
This module can be seen as the gradient of Conv2d with respect to
   its input. It is also known as a fractionally-strided convolution
   or a deconvolution (although it is not an actual deconvolution
   operation as it does not compute a true inverse of convolution).
   For more information, see the visualizations here and the
   Deconvolutional Networks paper.
This module supports TensorFloat32.
On certain ROCm devices, when using float16 inputs this module will
   use different precision for backward.


"stride" controls the stride for the cross-correlation.


"padding" controls the amount of implicit zero padding on both
sides for "dilation * (kernel_size - 1) - padding" number of
     points. See note below for details.


"output_padding" controls the additional size added to one side
     of the output shape. See note below for details.


"dilation" controls the spacing between the kernel points; also
     known as the Ã  trous algorithm. It is harder to describe, but the
     link here has a nice visualization of what "dilation" does.


"groups" controls the connections between inputs and outputs.
     "in_channels" and "out_channels" must both be divisible by
     "groups". For example,
* At groups=1, all inputs are convolved to all outputs.

* At groups=2, the operation becomes equivalent to having two
  conv layers side by side, each seeing half the input
  channels and producing half the output channels, and both
  subsequently concatenated.

* At groups= "in_channels", each input channel is convolved
  with its own set of filters (of size
with its own set of filters (of size
          \frac{\text{out_channels}}{\text{in_channels}}).
The parameters "kernel_size", "stride", "padding", "output_padding"
   can either be:
  * a single "int" -- in which case the same value is used for the
    height and width dimensions

  * a "tuple" of two ints -- in which case, the first *int* is
    used for the height dimension, and the second *int* for the
    width dimension

Note:
 The "padding" argument effectively adds "dilation * (kernel_size
 - 1) - padding" amount of zero padding to both sizes of the
 input. This is set so that when a "Conv2d" and a
 "ConvTranspose2d" are initialized with same parameters, they are
 inverses of each other in regard to the input and output shapes.
 However, when "stride > 1", "Conv2d" maps multiple input shapes
 to the same output shape. "output_padding" is provided to resolve
 this ambiguity by effectively increasing the calculated output
shape on one side. Note that "output_padding" is only used to
     find output shape, but does not actually add zero-padding to
     output.
Note:
 In some circumstances when given tensors on a CUDA device and
 using CuDNN, this operator may select a nondeterministic
 algorithm to increase performance. If this is undesirable, you
 can try to make the operation deterministic (potentially at a
 performance cost) by setting "torch.backends.cudnn.deterministic
 = True". See Reproducibility for more information.

Parameters:
      * in_channels (int) -- Number of channels in the input
        image
  * **out_channels** (*int*) -- Number of channels produced by the
    convolution

  * **kernel_size** (*int** or **tuple*) -- Size of the convolving
    kernel

  * **stride** (*int** or **tuple**, **optional*) -- Stride of the
    convolution. Default: 1

  * **padding** (*int** or **tuple**, **optional*) -- "dilation *
(kernel_size - 1) - padding" zero-padding will be added to
        both sides of each dimension in the input. Default: 0
  * **output_padding** (*int** or **tuple**, **optional*) --
    Additional size added to one side of each dimension in the
    output shape. Default: 0

  * **groups** (*int**, **optional*) -- Number of blocked
    connections from input channels to output channels. Default: 1

  * **bias** (*bool**, **optional*) -- If "True", adds a learnable
    bias to the output. Default: "True"

  * **dilation** (*int** or **tuple**, **optional*) -- Spacing
    between kernel elements. Default: 1

Shape:
      * Input: (N, C_{in}, H_{in}, W_{in}) or (C_{in}, H_{in}, W_{in})
  * Output: (N, C_{out}, H_{out}, W_{out}) or (C_{out}, H_{out},
    W_{out}), where

     H_{out} = (H_{in} - 1) \times \text{stride}[0] - 2 \times
     \text{padding}[0] + \text{dilation}[0]           \times
(\text{kernel_size}[0] - 1) + \text{output_padding}[0] + 1
     W_{out} = (W_{in} - 1) \times \text{stride}[1] - 2 \times
     \text{padding}[1] + \text{dilation}[1]           \times
     (\text{kernel\_size}[1] - 1) + \text{output\_padding}[1] + 1

Variables:
      * weight (Tensor) -- the learnable weights of the module
        of shape (\text{in_channels},
        \frac{\text{out_channels}}{\text{groups}},
        \text{kernel_size[0]}, \text{kernel_size[1]}). The values of
        these weights are sampled from \mathcal{U}(-\sqrt{k},
        \sqrt{k}) where k = \frac{groups}{C_\text{out} *
        \prod_{i=0}^{1}\text{kernel_size}[i]}
  * **bias** (*Tensor*) -- the learnable bias of the module of
    shape (out_channels) If "bias" is "True", then the values of
    these weights are sampled from \mathcal{U}(-\sqrt{k},
    \sqrt{k}) where k = \frac{groups}{C_\text{out} *
    \prod_{i=0}^{1}\text{kernel\_size}[i]}

Examples:
Examples:
  >>> # With square kernels and equal stride
  >>> m = nn.ConvTranspose2d(16, 33, 3, stride=2)
  >>> # non-square kernels and unequal stride and with padding
  >>> m = nn.ConvTranspose2d(16, 33, (3, 5), stride=(2, 1), padding=(4, 2))
  >>> input = torch.randn(20, 16, 50, 100)
  >>> output = m(input)
  >>> # exact output size can be also specified as an argument
  >>> input = torch.randn(1, 16, 12, 12)
  >>> downsample = nn.Conv2d(16, 16, 3, stride=2, padding=1)
  >>> upsample = nn.ConvTranspose2d(16, 16, 3, stride=2, padding=1)
  >>> h = downsample(input)
  >>> h.size()
  torch.Size([1, 16, 6, 6])
  >>> output = upsample(h, output_size=input.size())
  >>> output.size()
  torch.Size([1, 16, 12, 12])
torch.cuda.comm.reduce_add
torch.cuda.comm.reduce_add(inputs, destination=None)
Sums tensors from multiple GPUs.
All inputs should have matching shapes, dtype, and layout. The
   output tensor will be of the same shape, dtype, and layout.
Parameters:
      * inputs (Iterable[Tensor]) -- an iterable of
        tensors to add.
  * **destination** (*int**, **optional*) -- a device on which the
    output will be placed (default: current device).

Returns:
      A tensor containing an elementwise sum of all inputs, placed on
      the "destination" device.
torch.autograd.functional.hvp
torch.autograd.functional.hvp(func, inputs, v=None, create_graph=False, strict=False)
Function that computes the dot product between the Hessian of a
   given scalar function and a vector "v" at the point given by the
   inputs.
Parameters:
      * func (function) -- a Python function that takes Tensor
        inputs and returns a Tensor with a single element.
  * **inputs** (*tuple of Tensors** or **Tensor*) -- inputs to the
    function "func".

  * **v** (*tuple of Tensors** or **Tensor*) -- The vector for
    which the Hessian vector product is computed. Must be the same
    size as the input of "func". This argument is optional when
    "func"'s input contains a single element and (if it is not
    provided) will be set as a Tensor containing a single "1".

  * **create_graph** (*bool**, **optional*) -- If "True", both the
    output and result will be computed in a differentiable way.
Note that when "strict" is "False", the result can not require
        gradients or be disconnected from the inputs.  Defaults to
        "False".
  * **strict** (*bool**, **optional*) -- If "True", an error will
    be raised when we detect that there exists an input such that
    all the outputs are independent of it. If "False", we return a
    Tensor of zeros as the hvp for said inputs, which is the
    expected mathematical value. Defaults to "False".

Returns:
      tuple with:
         func_output (tuple of Tensors or Tensor): output of
         "func(inputs)"
     hvp (tuple of Tensors or Tensor): result of the dot product
     with the same shape as the inputs.

Return type:
      output (tuple)
-[ Example ]-



def pow_reducer(x):
   ...     return x.pow(3).sum()
inputs = torch.rand(2, 2)
v = torch.ones(2, 2)
hvp(pow_reducer, inputs, v)
   (tensor(0.1448),
    tensor([[2.0239, 1.6456],
            [2.4988, 1.4310]]))
[2.4988, 1.4310]]))



hvp(pow_reducer, inputs, v, create_graph=True)
   (tensor(0.1448, grad_fn=),
    tensor([[2.0239, 1.6456],
            [2.4988, 1.4310]], grad_fn=))
def pow_adder_reducer(x, y):
   ...     return (2 * x.pow(2) + 3 * y.pow(2)).sum()
inputs = (torch.rand(2), torch.rand(2))
v = (torch.zeros(2), torch.ones(2))
hvp(pow_adder_reducer, inputs, v)
   (tensor(2.3030),
    (tensor([0., 0.]),
     tensor([6., 6.])))



Note:
 This function is significantly slower than *vhp* due to backward
 mode AD constraints. If your functions is twice continuously
 differentiable, then hvp = vhp.t(). So if you know that your
 function satisfies this condition, you should use vhp instead
 that is much faster with the current implementation.
torch.nn.functional.conv2d
torch.nn.functional.conv2d(input, weight, bias=None, stride=1, padding=0, dilation=1, groups=1) -> Tensor
Applies a 2D convolution over an input image composed of several
   input planes.
This operator supports TensorFloat32.
See "Conv2d" for details and output shape.
Note:
 In some circumstances when given tensors on a CUDA device and
 using CuDNN, this operator may select a nondeterministic
 algorithm to increase performance. If this is undesirable, you
 can try to make the operation deterministic (potentially at a
 performance cost) by setting "torch.backends.cudnn.deterministic
 = True". See Reproducibility for more information.

Note:
 This operator supports complex data types i.e. "complex32,
 complex64, complex128".

Parameters:
      * input -- input tensor of shape (\text{minibatch} ,
        \text{in_channels} , iH , iW)
  * **weight** -- filters of shape (\text{out\_channels} ,
\frac{\text{in_channels}}{\text{groups}} , kH , kW)
  * **bias** -- optional bias tensor of shape
    (\text{out\_channels}). Default: "None"

  * **stride** -- the stride of the convolving kernel. Can be a
    single number or a tuple *(sH, sW)*. Default: 1

  * **padding** --

    implicit paddings on both sides of the input. Can be a string
    {'valid', 'same'}, single number or a tuple *(padH, padW)*.
    Default: 0 "padding='valid'" is the same as no padding.
    "padding='same'" pads the input so the output has the same
    shape as the input. However, this mode doesn't support any
    stride values other than 1.

    Warning:

      For "padding='same'", if the "weight" is even-length and
      "dilation" is odd in any dimension, a full "pad()" operation
      may be needed internally. Lowering performance.

  * **dilation** -- the spacing between kernel elements. Can be a
    single number or a tuple *(dH, dW)*. Default: 1
groups -- split input into groups, \text{in_channels}
        should be divisible by the number of groups. Default: 1

Examples:
  >>> # With square kernels and equal stride
  >>> filters = torch.randn(8, 4, 3, 3)
  >>> inputs = torch.randn(1, 4, 5, 5)
  >>> F.conv2d(inputs, filters, padding=1)
torch.func.vmap
torch.func.vmap(func, in_dims=0, out_dims=0, randomness='error', *, chunk_size=None)
vmap is the vectorizing map; "vmap(func)" returns a new function
   that maps "func" over some dimension of the inputs. Semantically,
   vmap pushes the map into PyTorch operations called by "func",
   effectively vectorizing those operations.
vmap is useful for handling batch dimensions: one can write a
   function "func" that runs on examples and then lift it to a
   function that can take batches of examples with "vmap(func)". vmap
   can also be used to compute batched gradients when composed with
   autograd.
Note:
 "torch.vmap()" is aliased to "torch.func.vmap()" for convenience.
 Use whichever one you'd like.

Parameters:
      * func (function) -- A Python function that takes one or
        more arguments. Must return one or more Tensors.
  * **in_dims** (*int** or **nested structure*) -- Specifies which
dimension of the inputs should be mapped over. "in_dims"
        should have a structure like the inputs. If the "in_dim" for a
        particular input is None, then that indicates there is no map
        dimension. Default: 0.
  * **out_dims** (*int** or **Tuple**[**int**]*) -- Specifies
    where the mapped dimension should appear in the outputs. If
    "out_dims" is a Tuple, then it should have one element per
    output. Default: 0.

  * **randomness** (*str*) -- Specifies whether the randomness in
    this vmap should be the same or different across batches. If
    'different', the randomness for each batch will be different.
    If 'same', the randomness will be the same across batches. If
    'error', any calls to random functions will error. Default:
    'error'. WARNING: this flag only applies to random PyTorch
    operations and does not apply to Python's random module or
    numpy randomness.
numpy randomness.
  * **chunk_size** (*None** or **int*) -- If None (default), apply
    a single vmap over inputs. If not None, then compute the vmap
    "chunk_size" samples at a time. Note that "chunk_size=1" is
    equivalent to computing the vmap with a for-loop. If you run
    into memory issues computing the vmap, please try a non-None
    chunk_size.

Returns:
      Returns a new "batched" function. It takes the same inputs as
      "func", except each input has an extra dimension at the index
      specified by "in_dims". It takes returns the same outputs as
      "func", except each output has an extra dimension at the index
      specified by "out_dims".
Return type:
      Callable
One example of using "vmap()" is to compute batched dot products.
   PyTorch doesn't provide a batched "torch.dot" API; instead of
   unsuccessfully rummaging through docs, use "vmap()" to construct a
   new function.