shrinath-suresh/blogs-docs-splitted huggingface dataset:
Raises:
      TypeError -- if "PRUNING_TYPE != 'unstructured'"
Note:
 Since global structured pruning doesn't make much sense unless
 the norm is normalized by the size of the parameter, we now limit
 the scope of global pruning to unstructured methods.

-[ Examples ]-



from torch.nn.utils import prune
from collections import OrderedDict
net = nn.Sequential(OrderedDict([
   ...     ('first', nn.Linear(10, 4)),
   ...     ('second', nn.Linear(4, 1)),
   ... ]))
parameters_to_prune = (
   ...     (net.first, 'weight'),
   ...     (net.second, 'weight'),
   ... )
prune.global_unstructured(
   ...     parameters_to_prune,
   ...     pruning_method=prune.L1Unstructured,
   ...     amount=10,
   ... )
print(sum(torch.nn.utils.parameters_to_vector(net.buffers()) == 0))
   tensor(10)
TransformerDecoderLayer
class torch.nn.TransformerDecoderLayer(d_model, nhead, dim_feedforward=2048, dropout=0.1, activation=, layer_norm_eps=1e-05, batch_first=False, norm_first=False, device=None, dtype=None)
TransformerDecoderLayer is made up of self-attn, multi-head-attn
   and feedforward network. This standard decoder layer is based on
   the paper "Attention Is All You Need". Ashish Vaswani, Noam
   Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez,
   Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you
   need. In Advances in Neural Information Processing Systems, pages
   6000-6010. Users may modify or implement in a different way during
   application.
Parameters:
      * d_model (int) -- the number of expected features in the
        input (required).
  * **nhead** (*int*) -- the number of heads in the
    multiheadattention models (required).

  * **dim_feedforward** (*int*) -- the dimension of the
feedforward network model (default=2048).
  * **dropout** (*float*) -- the dropout value (default=0.1).

  * **activation** (*Union**[**str**,
    **Callable**[**[**Tensor**]**, **Tensor**]**]*) -- the
    activation function of the intermediate layer, can be a string
    ("relu" or "gelu") or a unary callable. Default: relu

  * **layer_norm_eps** (*float*) -- the eps value in layer
    normalization components (default=1e-5).

  * **batch_first** (*bool*) -- If "True", then the input and
    output tensors are provided as (batch, seq, feature). Default:
    "False" (seq, batch, feature).

  * **norm_first** (*bool*) -- if "True", layer norm is done prior
    to self attention, multihead attention and feedforward
    operations, respectively. Otherwise it's done after. Default:
    "False" (after).

Examples::
      >>> decoder_layer = nn.TransformerDecoderLayer(d_model=512, nhead=8)
      >>> memory = torch.rand(10, 32, 512)
memory = torch.rand(10, 32, 512)
      >>> tgt = torch.rand(20, 32, 512)
      >>> out = decoder_layer(tgt, memory)



Alternatively, when "batch_first" is "True":
      >>> decoder_layer = nn.TransformerDecoderLayer(d_model=512, nhead=8, batch_first=True)
      >>> memory = torch.rand(32, 10, 512)
      >>> tgt = torch.rand(32, 20, 512)
      >>> out = decoder_layer(tgt, memory)
forward(tgt, memory, tgt_mask=None, memory_mask=None, tgt_key_padding_mask=None, memory_key_padding_mask=None, tgt_is_causal=False, memory_is_causal=False)
  Pass the inputs (and mask) through the decoder layer.

  Parameters:
     * **tgt** (*Tensor*) -- the sequence to the decoder layer
       (required).

     * **memory** (*Tensor*) -- the sequence from the last layer
       of the encoder (required).

     * **tgt_mask** (*Optional**[**Tensor**]*) -- the mask for the
       tgt sequence (optional).

     * **memory_mask** (*Optional**[**Tensor**]*) -- the mask for
the memory sequence (optional).
     * **tgt_key_padding_mask** (*Optional**[**Tensor**]*) -- the
       mask for the tgt keys per batch (optional).

     * **memory_key_padding_mask** (*Optional**[**Tensor**]*) --
       the mask for the memory keys per batch (optional).

     * **tgt_is_causal** (*bool*) -- If specified, applies a
       causal mask as tgt mask. Mutually exclusive with providing
       tgt_mask. Default: "False".

     * **memory_is_causal** (*bool*) -- If specified, applies a
       causal mask as tgt mask. Mutually exclusive with providing
       memory_mask. Default: "False".

  Return type:
     *Tensor*

  Shape:
     see the docs in Transformer class.
torch.randint_like
torch.randint_like(input, low=0, high, *, dtype=None, layout=torch.strided, device=None, requires_grad=False, memory_format=torch.preserve_format) -> Tensor
Returns a tensor with the same shape as Tensor "input" filled with
   random integers generated uniformly between "low" (inclusive) and
   "high" (exclusive).
Parameters:
      * input (Tensor) -- the size of "input" will determine
        size of the output tensor.
  * **low** (*int**, **optional*) -- Lowest integer to be drawn
    from the distribution. Default: 0.

  * **high** (*int*) -- One above the highest integer to be drawn
    from the distribution.

Keyword Arguments:
      * dtype ("torch.dtype", optional) -- the desired data type
        of returned Tensor. Default: if "None", defaults to the dtype
        of "input".
  * **layout** ("torch.layout", optional) -- the desired layout of
    returned tensor. Default: if "None", defaults to the layout of
torch.fft.fftfreq
torch.fft.fftfreq(n, d=1.0, *, out=None, dtype=None, layout=torch.strided, device=None, requires_grad=False) -> Tensor
Computes the discrete Fourier Transform sample frequencies for a
   signal of size "n".
Note:
 By convention, "fft()" returns positive frequency terms first,
 followed by the negative frequencies in reverse order, so that
 "f[-i]" for all 0 < i \leq n/2` in Python gives the negative
 frequency terms. For an FFT of length "n" and with inputs spaced
 in length unit "d", the frequencies are:

    f = [0, 1, ..., (n - 1) // 2, -(n // 2), ..., -1] / (d * n)

Note:
 For even lengths, the Nyquist frequency at "f[n/2]" can be
 thought of as either negative or positive. "fftfreq()" follows
 NumPy's convention of taking it to be negative.

Parameters:
      * n (int) -- the FFT length
  * **d** (*float**, **optional*) -- The sampling length scale.
The spacing between individual samples of the FFT input. The
        default assumes unit spacing, dividing that result by the
        actual spacing gives the result in physical frequency units.
Keyword Arguments:
      * out (Tensor, optional) -- the output tensor.
  * **dtype** ("torch.dtype", optional) -- the desired data type
    of returned tensor. Default: if "None", uses a global default
    (see "torch.set_default_tensor_type()").

  * **layout** ("torch.layout", optional) -- the desired layout of
    returned Tensor. Default: "torch.strided".

  * **device** ("torch.device", optional) -- the desired device of
    returned tensor. Default: if "None", uses the current device
    for the default tensor type (see
    "torch.set_default_tensor_type()"). "device" will be the CPU
    for CPU tensor types and the current CUDA device for CUDA
    tensor types.

  * **requires_grad** (*bool**, **optional*) -- If autograd should
record operations on the returned tensor. Default: "False".
-[ Example ]-



torch.fft.fftfreq(5)
   tensor([ 0.0000,  0.2000,  0.4000, -0.4000, -0.2000])



For even input, we can see the Nyquist frequency at "f[2]" is given
   as negative:



torch.fft.fftfreq(4)
   tensor([ 0.0000,  0.2500, -0.5000, -0.2500])
GELU
class torch.nn.GELU(approximate='none')
Applies the Gaussian Error Linear Units function:
  \text{GELU}(x) = x * \Phi(x)

where \Phi(x) is the Cumulative Distribution Function for Gaussian
   Distribution.
When the approximate argument is 'tanh', Gelu is estimated with:
  \text{GELU}(x) = 0.5 * x * (1 + \text{Tanh}(\sqrt(2 / \pi) * (x
  + 0.044715 * x^3)))

Parameters:
      approximate (str, optional) -- the gelu approximation
      algorithm to use: "'none'" | "'tanh'". Default: "'none'"
Shape:
      * Input: (*), where * means any number of dimensions.
  * Output: (*), same shape as the input.

[image]
Examples:
  >>> m = nn.GELU()
  >>> input = torch.randn(2)
  >>> output = m(input)
torch.func.functionalize
torch.func.functionalize(func, *, remove='mutations')
functionalize is a transform that can be used to remove
   (intermediate) mutations and aliasing from a function, while
   preserving the function's semantics.
"functionalize(func)" returns a new function with the same
   semantics as "func", but with all intermediate mutations removed.
   Every inplace operation performed on an intermediate tensor:
   "intermediate.foo_()" gets replaced by its out-of-place equivalent:
   "intermediate_updated = intermediate.foo()".
functionalize is useful for shipping a pytorch program off to
   backends or compilers that aren't able to easily represent
   mutations or aliasing operators.
Parameters:
      * func (Callable) -- A Python function that takes one or
        more arguments.
  * **remove** (*str*) -- An optional string argument, that takes
    on either the value 'mutations' or 'mutations_and_views'. If
'mutations' is passed in then all mutating operators will be
        replaced with their non-mutating equivalents. If
        'mutations_and_views' is passed in, then additionally, all
        aliasing operators will be replaced with their non-aliasing
        equivalents. Default: 'mutations'.
Returns:
      Returns a new "functionalized" function. It takes the same
      inputs as "func", and has the same behavior, but any mutations
      (and optionally aliasing) performed on intermeidate tensors in
      the function will be removed.
Return type:
      Callable
functionalize will also remove mutations (and views) that were
   performed on function inputs. However to preserve semantics,
   functionalize will "fix up" the mutations after the transform has
   finished running, by detecting if any tensor inputs "should have"
   been mutated, and copying the new data back to the inputs if
   necessary.
Example:
  >>> import torch
necessary.
Example:
  >>> import torch
  >>> from torch.fx.experimental.proxy_tensor import make_fx
  >>> from torch.func import functionalize
  >>>
  >>> # A function that uses mutations and views, but only on intermediate tensors.
  >>> def f(a):
  ...     b = a + 1
  ...     c = b.view(-1)
  ...     c.add_(1)
  ...     return b
  ...
  >>> inpt = torch.randn(2)
  >>>
  >>> out1 = f(inpt)
  >>> out2 = functionalize(f)(inpt)
  >>>
  >>> # semantics are the same (outputs are equivalent)
  >>> print(torch.allclose(out1, out2))
  True
  >>>
  >>> f_traced = make_fx(f)(inpt)
  >>> f_no_mutations_traced = make_fx(functionalize(f))(inpt)
  >>> f_no_mutations_and_views_traced = make_fx(functionalize(f, remove='mutations_and_views'))(inpt)
  >>>
  >>> print(f_traced.code)



  def forward(self, a_1):
      add = torch.ops.aten.add(a_1, 1);  a_1 = None
      view = torch.ops.aten.view(add, [-1])
view = torch.ops.aten.view(add, [-1])
          add_ = torch.ops.aten.add_(view, 1);  view = None
          return add
  >>> print(f_no_mutations_traced.code)



  def forward(self, a_1):
      add = torch.ops.aten.add(a_1, 1);  a_1 = None
      view = torch.ops.aten.view(add, [-1]);  add = None
      add_1 = torch.ops.aten.add(view, 1);  view = None
      view_1 = torch.ops.aten.view(add_1, [2]);  add_1 = None
      return view_1

  >>> print(f_no_mutations_and_views_traced.code)



  def forward(self, a_1):
      add = torch.ops.aten.add(a_1, 1);  a_1 = None
      view_copy = torch.ops.aten.view_copy(add, [-1]);  add = None
      add_1 = torch.ops.aten.add(view_copy, 1);  view_copy = None
      view_copy_1 = torch.ops.aten.view_copy(add_1, [2]);  add_1 = None
      return view_copy_1


  >>> # A function that mutates its input tensor
  >>> def f(a):
  ...     b = a.view(-1)
  ...     b.add_(1)
  ...     return a
  ...
...     return a
      ...
      >>> f_no_mutations_and_views_traced = make_fx(functionalize(f, remove='mutations_and_views'))(inpt)
      >>> #
      >>> # All mutations and views have been removed,
      >>> # but there is an extra copy_ in the graph to correctly apply the mutation to the input
      >>> # after the function has completed.
      >>> print(f_no_mutations_and_views_traced.code)
  def forward(self, a_1):
      view_copy = torch.ops.aten.view_copy(a_1, [-1])
      add = torch.ops.aten.add(view_copy, 1);  view_copy = None
      view_copy_1 = torch.ops.aten.view_copy(add, [2]);  add = None
      copy_ = torch.ops.aten.copy_(a_1, view_copy_1);  a_1 = None
      return view_copy_1

There are a few "failure modes" for functionalize that are worth
   calling out:
      1. Like other torch.func transforms, functionalize() doesn't
         work with functions that directly use .backward(). The same
is true for torch.autograd.grad. If you want to use autograd,
         you can compute gradients directly with
         functionalize(grad(f)).
  2. Like other torch.func transforms, *functionalize()* doesn't
     work with global state. If you call *functionalize(f)* on a
     function that takes views / mutations of non-local state,
     functionalization will simply no-op and pass the
     view/mutation calls directly to the backend. One way to work
     around this is is to ensure that any non-local state creation
     is wrapped into a larger function, which you then call
     functionalize on.

  3. *resize_()* has some limitations: functionalize will only
     work on programs that use resize_()` as long as the tensor
     being resized is not a view.

  4. *as_strided()* has some limitations: functionalize will not
     work on *as_strided()* calls that result in tensors with
     overlapping memory.
overlapping memory.
Finally, a helpful mental model for understanding functionalization
   is that most user pytorch programs are writting with the public
   torch API. When executed, torch operators are generally decomposed
   into our internal C++ "ATen" API. The logic for functionalization
   happens entirely at the level of ATen. Functionalization knows how
   to take every aliasing operator in ATen, and map it to its non-
   aliasing equivalent (e.g. "tensor.view({-1})" ->
   "at::view_copy(tensor, {-1})"), and how to take every mutating
   operator in ATen, and map it to its non-mutating equivalent (e.g.
   "tensor.add_(1)" -> "at::add(tensor, -1)"), while tracking aliases
   and mutations out-of-line to know when to fix things up.
   Information about which ATen operators are aliasing or mutating all
   comes from https://github.com/pytorch/pytorch/blob/master/aten/src
   /ATen/native/native_functions.yaml.
torch.bernoulli
torch.bernoulli(input, *, generator=None, out=None) -> Tensor
Draws binary random numbers (0 or 1) from a Bernoulli distribution.
The "input" tensor should be a tensor containing probabilities to
   be used for drawing the binary random number. Hence, all values in
   "input" have to be in the range: 0 \leq \text{input}_i \leq 1.
The \text{i}^{th} element of the output tensor will draw a value 1
   according to the \text{i}^{th} probability value given in "input".
  \text{out}_{i} \sim \mathrm{Bernoulli}(p = \text{input}_{i})

The returned "out" tensor only has values 0 or 1 and is of the same
   shape as "input".
"out" can have integral "dtype", but "input" must have floating
   point "dtype".
Parameters:
      input (Tensor) -- the input tensor of probability values
      for the Bernoulli distribution
Keyword Arguments:
      * generator ("torch.Generator", optional) -- a pseudorandom
        number generator for sampling
number generator for sampling
  * **out** (*Tensor**, **optional*) -- the output tensor.

Example:
  >>> a = torch.empty(3, 3).uniform_(0, 1)  # generate a uniform random matrix with range [0, 1]
  >>> a
  tensor([[ 0.1737,  0.0950,  0.3609],
          [ 0.7148,  0.0289,  0.2676],
          [ 0.9456,  0.8937,  0.7202]])
  >>> torch.bernoulli(a)
  tensor([[ 1.,  0.,  0.],
          [ 0.,  0.,  0.],
          [ 1.,  1.,  1.]])

  >>> a = torch.ones(3, 3) # probability of drawing "1" is 1
  >>> torch.bernoulli(a)
  tensor([[ 1.,  1.,  1.],
          [ 1.,  1.,  1.],
          [ 1.,  1.,  1.]])
  >>> a = torch.zeros(3, 3) # probability of drawing "1" is 0
  >>> torch.bernoulli(a)
  tensor([[ 0.,  0.,  0.],
          [ 0.,  0.,  0.],
          [ 0.,  0.,  0.]])
torch.minimum
torch.minimum(input, other, *, out=None) -> Tensor
Computes the element-wise minimum of "input" and "other".
Note:
 If one of the elements being compared is a NaN, then that element
 is returned. "minimum()" is not supported for tensors with
 complex dtypes.

Parameters:
      * input (Tensor) -- the input tensor.
  * **other** (*Tensor*) -- the second input tensor

Keyword Arguments:
      out (Tensor, optional) -- the output tensor.
Example:
  >>> a = torch.tensor((1, 2, -1))
  >>> b = torch.tensor((3, 0, 4))
  >>> torch.minimum(a, b)
  tensor([1, 0, -1])
torch.logical_and
torch.logical_and(input, other, *, out=None) -> Tensor
Computes the element-wise logical AND of the given input tensors.
   Zeros are treated as "False" and nonzeros are treated as "True".
Parameters:
      * input (Tensor) -- the input tensor.
  * **other** (*Tensor*) -- the tensor to compute AND with

Keyword Arguments:
      out (Tensor, optional) -- the output tensor.
Example:
  >>> torch.logical_and(torch.tensor([True, False, True]), torch.tensor([True, False, False]))
  tensor([ True, False, False])
  >>> a = torch.tensor([0, 1, 10, 0], dtype=torch.int8)
  >>> b = torch.tensor([4, 0, 1, 0], dtype=torch.int8)
  >>> torch.logical_and(a, b)
  tensor([False, False,  True, False])
  >>> torch.logical_and(a.double(), b.double())
  tensor([False, False,  True, False])
  >>> torch.logical_and(a.double(), b)
  tensor([False, False,  True, False])
CELU
class torch.nn.CELU(alpha=1.0, inplace=False)
Applies the element-wise function:
  \text{CELU}(x) = \max(0,x) + \min(0, \alpha * (\exp(x/\alpha) -
  1))

More details can be found in the paper Continuously Differentiable
   Exponential Linear Units .
Parameters:
      * alpha (float) -- the \alpha value for the CELU
        formulation. Default: 1.0
  * **inplace** (*bool*) -- can optionally do the operation in-
    place. Default: "False"

Shape:
      * Input: (*), where * means any number of dimensions.
  * Output: (*), same shape as the input.

[image]
Examples:
  >>> m = nn.CELU()
  >>> input = torch.randn(2)
  >>> output = m(input)
TransformerDecoder
class torch.nn.TransformerDecoder(decoder_layer, num_layers, norm=None)
TransformerDecoder is a stack of N decoder layers
Parameters:
      * decoder_layer -- an instance of the
        TransformerDecoderLayer() class (required).
  * **num_layers** -- the number of sub-decoder-layers in the
    decoder (required).

  * **norm** -- the layer normalization component (optional).

Examples::
      >>> decoder_layer = nn.TransformerDecoderLayer(d_model=512, nhead=8)
      >>> transformer_decoder = nn.TransformerDecoder(decoder_layer, num_layers=6)
      >>> memory = torch.rand(10, 32, 512)
      >>> tgt = torch.rand(20, 32, 512)
      >>> out = transformer_decoder(tgt, memory)
forward(tgt, memory, tgt_mask=None, memory_mask=None, tgt_key_padding_mask=None, memory_key_padding_mask=None)
  Pass the inputs (and mask) through the decoder layer in turn.

  Parameters:
     * **tgt** (*Tensor*) -- the sequence to the decoder
(required).
     * **memory** (*Tensor*) -- the sequence from the last layer
       of the encoder (required).

     * **tgt_mask** (*Optional**[**Tensor**]*) -- the mask for the
       tgt sequence (optional).

     * **memory_mask** (*Optional**[**Tensor**]*) -- the mask for
       the memory sequence (optional).

     * **tgt_key_padding_mask** (*Optional**[**Tensor**]*) -- the
       mask for the tgt keys per batch (optional).

     * **memory_key_padding_mask** (*Optional**[**Tensor**]*) --
       the mask for the memory keys per batch (optional).

  Return type:
     *Tensor*

  Shape:
     see the docs in Transformer class.
avg_pool3d
class torch.ao.nn.quantized.functional.avg_pool3d(input, kernel_size, stride=None, padding=0, ceil_mode=False, count_include_pad=True, divisor_override=None)
Applies 3D average-pooling operation in kD \ times kH \times kW
   regions by step size sD \times sH \times sW steps. The number of
   output features is equal to the number of input planes.
Note:
 The input quantization parameters propagate to the output.

Parameters:
      * input -- quantized input tensor (\text{minibatch} ,
        \text{in_channels} , iH , iW)
  * **kernel_size** -- size of the pooling region. Can be a single
    number or a tuple *(kD, kH, kW)*

  * **stride** -- stride of the pooling operation. Can be a single
    number or a tuple *(sD, sH, sW)*. Default: "kernel_size"

  * **padding** -- implicit zero paddings on both sides of the
    input. Can be a single number or a tuple *(padD, padH, padW)*.
    Default: 0
Default: 0
  * **ceil_mode** -- when True, will use *ceil* instead of *floor*
    in the formula to compute the output shape. Default: "False"

  * **count_include_pad** -- when True, will include the zero-
    padding in the averaging calculation. Default: "True"

  * **divisor_override** -- if specified, it will be used as
    divisor, otherwise size of the pooling region will be used.
    Default: None
torch.nn.functional.dropout
torch.nn.functional.dropout(input, p=0.5, training=True, inplace=False)
During training, randomly zeroes some of the elements of the input
   tensor with probability "p" using samples from a Bernoulli
   distribution.
See "Dropout" for details.
Parameters:
      * p (float) -- probability of an element to be zeroed.
        Default: 0.5
  * **training** (*bool*) -- apply dropout if is "True". Default:
    "True"

  * **inplace** (*bool*) -- If set to "True", will do this
    operation in-place. Default: "False"

Return type:
      Tensor
torch.Tensor.expand
Tensor.expand(*sizes) -> Tensor
Returns a new view of the "self" tensor with singleton dimensions
   expanded to a larger size.
Passing -1 as the size for a dimension means not changing the size
   of that dimension.
Tensor can be also expanded to a larger number of dimensions, and
   the new ones will be appended at the front. For the new dimensions,
   the size cannot be set to -1.
Expanding a tensor does not allocate new memory, but only creates a
   new view on the existing tensor where a dimension of size one is
   expanded to a larger size by setting the "stride" to 0. Any
   dimension of size 1 can be expanded to an arbitrary value without
   allocating new memory.
Parameters:
      sizes (torch.Size or int...*) -- the desired
      expanded size
Warning:
 More than one element of an expanded tensor may refer to a single
 memory location. As a result, in-place operations (especially
ones that are vectorized) may result in incorrect behavior. If
     you need to write to the tensors, please clone them first.
Example:
  >>> x = torch.tensor([[1], [2], [3]])
  >>> x.size()
  torch.Size([3, 1])
  >>> x.expand(3, 4)
  tensor([[ 1,  1,  1,  1],
          [ 2,  2,  2,  2],
          [ 3,  3,  3,  3]])
  >>> x.expand(-1, 4)   # -1 means not changing the size of that dimension
  tensor([[ 1,  1,  1,  1],
          [ 2,  2,  2,  2],
          [ 3,  3,  3,  3]])
RReLU
class torch.nn.RReLU(lower=0.125, upper=0.3333333333333333, inplace=False)
Applies the randomized leaky rectified liner unit function,
   element-wise, as described in the paper:
Empirical Evaluation of Rectified Activations in Convolutional
   Network.
The function is defined as:
  \text{RReLU}(x) = \begin{cases}     x & \text{if } x \geq 0 \\
  ax & \text{ otherwise } \end{cases}

where a is randomly sampled from uniform distribution
   \mathcal{U}(\text{lower}, \text{upper}).
  See: https://arxiv.org/pdf/1505.00853.pdf

Parameters:
      * lower (float) -- lower bound of the uniform
        distribution. Default: \frac{1}{8}
  * **upper** (*float*) -- upper bound of the uniform
    distribution. Default: \frac{1}{3}

  * **inplace** (*bool*) -- can optionally do the operation in-
    place. Default: "False"

Shape:
      * Input: (*), where * means any number of dimensions.
  * Output: (*), same shape as the input.

[image]