shrinath-suresh/blogs-docs-splitted huggingface dataset:
torch.special.ndtri(input, *, out=None) -> Tensor
Computes the argument, x, for which the area under the Gaussian
   probability density function (integrated from minus infinity to x)
   is equal to "input", elementwise.
  \text{ndtri}(p) = \sqrt{2}\text{erf}^{-1}(2p - 1)

Note:
 Also known as quantile function for Normal Distribution.

Parameters:
      input (Tensor) -- the input tensor.
Keyword Arguments:
      out (Tensor, optional) -- the output tensor.
Example::
      >>> torch.special.ndtri(torch.tensor([0, 0.25, 0.5, 0.75, 1]))
      tensor([   -inf, -0.6745,  0.0000,  0.6745,     inf])
torch.special.polygamma(n, input, *, out=None) -> Tensor
Computes the n^{th} derivative of the digamma function on "input".
   n \geq 0 is called the order of the polygamma function.
  \psi^{(n)}(x) = \frac{d^{(n)}}{dx^{(n)}} \psi(x)

Note:
 This function is implemented only for nonnegative integers n \geq
 0.

Parameters:
0.
Parameters:
      * n (int) -- the order of the polygamma function
  * **input** (*Tensor*) -- the input tensor.

Keyword Arguments:
      out (Tensor, optional) -- the output tensor.
Example::
      >>> a = torch.tensor([1, 0.5])
      >>> torch.special.polygamma(1, a)
      tensor([1.64493, 4.9348])
      >>> torch.special.polygamma(2, a)
      tensor([ -2.4041, -16.8288])
      >>> torch.special.polygamma(3, a)
      tensor([ 6.4939, 97.4091])
      >>> torch.special.polygamma(4, a)
      tensor([ -24.8863, -771.4742])
torch.special.psi(input, *, out=None) -> Tensor
Alias for "torch.special.digamma()".
torch.special.round(input, *, out=None) -> Tensor
Alias for "torch.round()".
torch.special.scaled_modified_bessel_k0(input, *, out=None) -> Tensor
Scaled modified Bessel function of the second kind of order 0.
Parameters:
      input (Tensor) -- the input tensor.
Keyword Arguments:
Keyword Arguments:
      out (Tensor, optional) -- the output tensor.
torch.special.scaled_modified_bessel_k1(input, *, out=None) -> Tensor
Scaled modified Bessel function of the second kind of order 1.
Parameters:
      input (Tensor) -- the input tensor.
Keyword Arguments:
      out (Tensor, optional) -- the output tensor.
torch.special.sinc(input, *, out=None) -> Tensor
Computes the normalized sinc of "input."
  \text{out}_{i} = \begin{cases}   1, & \text{if}\
  \text{input}_{i}=0 \\   \sin(\pi \text{input}_{i}) / (\pi
  \text{input}_{i}), & \text{otherwise} \end{cases}

Parameters:
      input (Tensor) -- the input tensor.
Keyword Arguments:
      out (Tensor, optional) -- the output tensor.
Example::
      >>> t = torch.randn(4)
      >>> t
      tensor([ 0.2252, -0.2948,  1.0267, -1.1566])
      >>> torch.special.sinc(t)
      tensor([ 0.9186,  0.8631, -0.0259, -0.1300])
torch.special.softmax(input, dim, *, dtype=None) -> Tensor
Computes the softmax function.
Softmax is defined as:
\text{Softmax}(x_{i}) = \frac{\exp(x_i)}{\sum_j \exp(x_j)}
It is applied to all slices along dim, and will re-scale them so
   that the elements lie in the range [0, 1] and sum to 1.
Parameters:
      * input (Tensor) -- input
  * **dim** (*int*) -- A dimension along which softmax will be
    computed.

  * **dtype** ("torch.dtype", optional) -- the desired data type
    of returned tensor. If specified, the input tensor is cast to
    "dtype" before the operation is performed. This is useful for
    preventing data type overflows. Default: None.

Examples::
      >>> t = torch.ones(2, 2)
      >>> torch.special.softmax(t, 0)
      tensor([[0.5000, 0.5000],
              [0.5000, 0.5000]])
torch.special.spherical_bessel_j0(input, *, out=None) -> Tensor
Spherical Bessel function of the first kind of order 0.
Parameters:
Parameters:
      input (Tensor) -- the input tensor.
Keyword Arguments:
      out (Tensor, optional) -- the output tensor.
torch.special.xlog1py(input, other, *, out=None) -> Tensor
Computes "input * log1p(other)" with the following cases.
  \text{out}_{i} = \begin{cases}     \text{NaN} & \text{if }
  \text{other}_{i} = \text{NaN} \\     0 & \text{if }
  \text{input}_{i} = 0.0 \text{ and } \text{other}_{i} !=
  \text{NaN} \\     \text{input}_{i} *
  \text{log1p}(\text{other}_{i})& \text{otherwise} \end{cases}

Similar to SciPy's scipy.special.xlog1py.
Parameters:
      * input (Number or Tensor) -- Multiplier
  * **other** (*Number** or **Tensor*) -- Argument

Note:
 At least one of "input" or "other" must be a tensor.

Keyword Arguments:
      out (Tensor, optional) -- the output tensor.
Example:
  >>> x = torch.zeros(5,)
  >>> y = torch.tensor([-1, 0, 1, float('inf'), float('nan')])
torch.special.xlog1py(x, y)
      tensor([0., 0., 0., 0., nan])
      >>> x = torch.tensor([1, 2, 3])
      >>> y = torch.tensor([3, 2, 1])
      >>> torch.special.xlog1py(x, y)
      tensor([1.3863, 2.1972, 2.0794])
      >>> torch.special.xlog1py(x, 4)
      tensor([1.6094, 3.2189, 4.8283])
      >>> torch.special.xlog1py(2, y)
      tensor([2.7726, 2.1972, 1.3863])



torch.special.xlogy(input, other, *, out=None) -> Tensor
Computes "input * log(other)" with the following cases.
  \text{out}_{i} = \begin{cases}     \text{NaN} & \text{if }
  \text{other}_{i} = \text{NaN} \\     0 & \text{if }
  \text{input}_{i} = 0.0 \\     \text{input}_{i} *
  \log{(\text{other}_{i})} & \text{otherwise} \end{cases}

Similar to SciPy's scipy.special.xlogy.
Parameters:
      * input (Number or Tensor) -- Multiplier
  * **other** (*Number** or **Tensor*) -- Argument

Note:
 At least one of "input" or "other" must be a tensor.

Keyword Arguments:
Keyword Arguments:
      out (Tensor, optional) -- the output tensor.
Example:
  >>> x = torch.zeros(5,)
  >>> y = torch.tensor([-1, 0, 1, float('inf'), float('nan')])
  >>> torch.special.xlogy(x, y)
  tensor([0., 0., 0., 0., nan])
  >>> x = torch.tensor([1, 2, 3])
  >>> y = torch.tensor([3, 2, 1])
  >>> torch.special.xlogy(x, y)
  tensor([1.0986, 1.3863, 0.0000])
  >>> torch.special.xlogy(x, 4)
  tensor([1.3863, 2.7726, 4.1589])
  >>> torch.special.xlogy(2, y)
  tensor([2.1972, 1.3863, 0.0000])

torch.special.zeta(input, other, *, out=None) -> Tensor
Computes the Hurwitz zeta function, elementwise.
  \zeta(x, q) = \sum_{k=0}^{\infty} \frac{1}{(k + q)^x}

Parameters:
      * input (Tensor) -- the input tensor corresponding to x.
  * **other** (*Tensor*) -- the input tensor corresponding to *q*.

Note:
 The Riemann zeta function corresponds to the case when *q = 1*

Keyword Arguments:
Keyword Arguments:
      out (Tensor, optional) -- the output tensor.
Example::
      >>> x = torch.tensor([2., 4.])
      >>> torch.special.zeta(x, 1)
      tensor([1.6449, 1.0823])
      >>> torch.special.zeta(x, torch.tensor([1., 2.]))
      tensor([1.6449, 0.0823])
      >>> torch.special.zeta(2, torch.tensor([1., 2.]))
      tensor([1.6449, 0.6449])
torch.utils.bottleneck
torch.utils.bottleneck is a tool that can be used as an initial step
for debugging bottlenecks in your program. It summarizes runs of your
script with the Python profiler and PyTorch's autograd profiler.
Run it on the command line with
python -m torch.utils.bottleneck /path/to/source/script.py [args]
where [args] are any number of arguments to script.py, or run
"python -m torch.utils.bottleneck -h" for more usage instructions.
Warning:
Because your script will be profiled, please ensure that it exits in
  a finite amount of time.
Warning:
Due to the asynchronous nature of CUDA kernels, when running against
  CUDA code, the cProfile output and CPU-mode autograd profilers may
  not show correct timings: the reported CPU time reports the amount
  of time used to launch the kernels but does not include the time the
  kernel spent executing on a GPU unless the operation does a
  synchronize. Ops that do synchronize appear to be extremely
expensive under regular CPU-mode profilers. In these case where
  timings are incorrect, the CUDA-mode autograd profiler may be
  helpful.
Note:
To decide which (CPU-only-mode or CUDA-mode) autograd profiler
  output to look at, you should first check if your script is CPU-
  bound ("CPU total time is much greater than CUDA total time"). If it
  is CPU-bound, looking at the results of the CPU-mode autograd
  profiler will help. If on the other hand your script spends most of
  its time executing on the GPU, then it makes sense to start looking
  for responsible CUDA operators in the output of the CUDA-mode
  autograd profiler.Of course the reality is much more complicated and
  your script might not be in one of those two extremes depending on
  the part of the model you're evaluating. If the profiler outputs
  don't help, you could try looking at the result of
  "torch.autograd.profiler.emit_nvtx()" with "nvprof". However, please
  take into account that the NVTX overhead is very high and often
gives a heavily skewed timeline. Similarly, "IntelÂ® VTuneâ¢ Profiler"
  helps to analyze performance on Intel platforms further with
  "torch.autograd.profiler.emit_itt()".
Warning:
If you are profiling CUDA code, the first profiler that "bottleneck"
  runs (cProfile) will include the CUDA startup time (CUDA buffer
  allocation cost) in its time reporting. This should not matter if
  your bottlenecks result in code much slower than the CUDA startup
  time.
For more complicated uses of the profilers (like in a multi-GPU case),
please see https://docs.python.org/3/library/profile.html or
"torch.autograd.profiler.profile()" for more information.
Frequently Asked Questions
My model reports "cuda runtime error(2): out of memory"
As the error message suggests, you have run out of memory on your GPU.
Since we often deal with large amounts of data in PyTorch, small
mistakes can rapidly cause your program to use up all of your GPU;
fortunately, the fixes in these cases are often simple. Here are a few
common things to check:
Don't accumulate history across your training loop. By default,
computations involving variables that require gradients will keep
history.  This means that you should avoid using such variables in
computations which will live beyond your training loops, e.g., when
tracking statistics. Instead, you should detach the variable or access
its underlying data.
Sometimes, it can be non-obvious when differentiable variables can
occur.  Consider the following training loop (abridged from source):
total_loss = 0
   for i in range(10000):
total_loss = 0
   for i in range(10000):
       optimizer.zero_grad()
       output = model(input)
       loss = criterion(output)
       loss.backward()
       optimizer.step()
       total_loss += loss
Here, "total_loss" is accumulating history across your training loop,
since "loss" is a differentiable variable with autograd history. You
can fix this by writing total_loss += float(loss) instead.
Other instances of this problem: 1.
Don't hold onto tensors and variables you don't need. If you
assign a Tensor or Variable to a local, Python will not deallocate
until the local goes out of scope.  You can free this reference by
using "del x".  Similarly, if you assign a Tensor or Variable to a
member variable of an object, it will not deallocate until the object
goes out of scope.  You will get the best memory usage if you don't
hold onto temporaries you don't need.
The scopes of locals can be larger than you expect.  For example:
for i in range(5):
       intermediate = f(input[i])
intermediate = f(input[i])
       result += g(intermediate)
   output = h(result)
   return output
Here, "intermediate" remains live even while "h" is executing, because
its scope extrudes past the end of the loop.  To free it earlier, you
should "del intermediate" when you are done with it.
Avoid running RNNs on sequences that are too large. The amount of
memory required to backpropagate through an RNN scales linearly with
the length of the RNN input; thus, you will run out of memory if you
try to feed an RNN a sequence that is too long.
The technical term for this phenomenon is backpropagation through
time, and there are plenty of references for how to implement
truncated BPTT, including in the word language model example;
truncation is handled by the "repackage" function as described in this
forum post.
Don't use linear layers that are too large. A linear layer
"nn.Linear(m, n)" uses O(nm) memory: that is to say, the memory
requirements of the weights scales quadratically with the number of
features.  It is very easy to blow through your memory this way (and
remember that you will need at least twice the size of the weights,
since you also need to store the gradients.)
Consider checkpointing. You can trade-off memory for compute by
using checkpoint.
My GPU memory isn't freed properly
PyTorch uses a caching memory allocator to speed up memory
allocations. As a result, the values shown in "nvidia-smi" usually
don't reflect the true memory usage. See Memory management for more
details about GPU memory management.
If your GPU memory isn't freed even after Python quits, it is very
likely that some Python subprocesses are still alive. You may find
them via "ps -elf | grep python" and manually kill them with "kill -9
[pid]".
My out of memory exception handler can't allocate memory
You may have some code that tries to recover from out of memory
errors.
try:
       run_model(batch_size)
   except RuntimeError: # Out of memory
       for _ in range(batch_size):
           run_model(1)
But find that when you do run out of memory, your recovery code can't
allocate either. That's because the python exception object holds a
reference to the stack frame where the error was raised. Which
prevents the original tensor objects from being freed. The solution is
to move you OOM recovery code outside of the "except" clause.
oom = False
   try:
       run_model(batch_size)
   except RuntimeError: # Out of memory
       oom = True
if oom:
       for _ in range(batch_size):
           run_model(1)
My data loader workers return identical random numbers
You are likely using other libraries to generate random numbers in the
dataset and worker subprocesses are started via "fork". See
"torch.utils.data.DataLoader"'s documentation for how to properly set
up random seeds in workers with its "worker_init_fn" option.
My recurrent network doesn't work with data parallelism
There is a subtlety in using the "pack sequence -> recurrent network
-> unpack sequence" pattern in a "Module" with "DataParallel" or
"data_parallel()". Input to each the "forward()" on each device will
only be part of the entire input. Because the unpack operation
"torch.nn.utils.rnn.pad_packed_sequence()" by default only pads up to
the longest input it sees, i.e., the longest on that particular
device, size mismatches will happen when results are gathered
together. Therefore, you can instead take advantage of the
"total_length" argument of "pad_packed_sequence()" to make sure that
the "forward()" calls return sequences of same length. For example,
you can write:
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence
class MyModule(nn.Module):
       # ... init, other methods, etc.
   # padded_input is of shape [B x T x *] (batch_first mode) and contains
   # the sequences sorted by lengths
   #   B is the batch size
   #   T is max sequence length
   def forward(self, padded_input, input_lengths):
       total_length = padded_input.size(1)  # get the max sequence length
       packed_input = pack_padded_sequence(padded_input, input_lengths,
                                           batch_first=True)
       packed_output, _ = self.my_lstm(packed_input)
       output, _ = pad_packed_sequence(packed_output, batch_first=True,
                                       total_length=total_length)
       return output

m = MyModule().cuda()
   dp_m = nn.DataParallel(m)
Additionally, extra care needs to be taken when batch dimension is dim
"1" (i.e., "batch_first=False") with data parallelism. In this case,
MPS backend
"mps" device enables high-performance training on GPU for MacOS
devices with Metal programming framework.  It introduces a new device
to map Machine Learning computational graphs and primitives on highly
efficient Metal Performance Shaders Graph framework and tuned kernels
provided by Metal Performance Shaders framework respectively.
The new MPS backend extends the PyTorch ecosystem and provides
existing scripts capabilities to setup and run operations on GPU.
To get started, simply move your Tensor and Module to the "mps"
device:
# Check that MPS is available
   if not torch.backends.mps.is_available():
       if not torch.backends.mps.is_built():
           print("MPS not available because the current PyTorch install was not "
                 "built with MPS enabled.")
       else:
           print("MPS not available because the current MacOS version is not 12.3+ "
                 "and/or you do not have an MPS-enabled device on this machine.")
else:
else:
       mps_device = torch.device("mps")
   # Create a Tensor directly on the mps device
   x = torch.ones(5, device=mps_device)
   # Or
   x = torch.ones(5, device="mps")

   # Any operation happens on the GPU
   y = x * 2

   # Move your model to mps just like any other device
   model = YourFavoriteNet()
   model.to(mps_device)

   # Now every call runs on the GPU
   pred = model(x)
Distributed Data Parallel
Warning:
The implementation of "torch.nn.parallel.DistributedDataParallel"
  evolves over time. This design note is written based on the state as
  of v1.4.
"torch.nn.parallel.DistributedDataParallel" (DDP) transparently
performs distributed data parallel training. This page describes how
it works and reveals implementation details.
Example
Let us start with a simple "torch.nn.parallel.DistributedDataParallel"
example. This example uses a "torch.nn.Linear" as the local model,
wraps it with DDP, and then runs one forward pass, one backward pass,
and an optimizer step on the DDP model. After that, parameters on the
local model will be updated, and all models on different processes
should be exactly the same.
import torch
   import torch.distributed as dist
   import torch.multiprocessing as mp
   import torch.nn as nn
   import torch.optim as optim
   from torch.nn.parallel import DistributedDataParallel as DDP
def example(rank, world_size):
       # create default process group
       dist.init_process_group("gloo", rank=rank, world_size=world_size)
       # create local model
       model = nn.Linear(10, 10).to(rank)
       # construct DDP model
       ddp_model = DDP(model, device_ids=[rank])
       # define loss function and optimizer
       loss_fn = nn.MSELoss()
       optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)
   # forward pass
   outputs = ddp_model(torch.randn(20, 10).to(rank))
   labels = torch.randn(20, 10).to(rank)
   # backward pass
   loss_fn(outputs, labels).backward()
   # update parameters
   optimizer.step()

def main():
       world_size = 2
       mp.spawn(example,
           args=(world_size,),
           nprocs=world_size,
           join=True)
if name=="main":
       # Environment variables which need to be
       # set when using c10d's default "env"
       # initialization mode.
       os.environ["MASTER_ADDR"] = "localhost"
os.environ["MASTER_ADDR"] = "localhost"
       os.environ["MASTER_PORT"] = "29500"
       main()
DDP works with TorchDynamo.  When used with TorchDynamo, apply the DDP
model wrapper before compiling the model, such that torchdynamo can
apply "DDPOptimizer" (graph-break optimizations) based on DDP bucket
sizes.  (See TorchDynamo DDPOptimizer for more information.)
TorchDynamo support for DDP currently requires setting
static_graph=False, due to interactions between the graph tracing
process and DDP's mechanism for observing operations happening on its
module, but this should be fixed ultimately.
ddp_model = DDP(model, device_ids=[rank])
   ddp_model = torch.compile(ddp_model)
Internal Design
This section reveals how it works under the hood of
"torch.nn.parallel.DistributedDataParallel" by diving into details of
every step in one iteration.

Prerequisite: DDP relies on c10d "ProcessGroup" for
  communications. Hence, applications must create "ProcessGroup"
instances before constructing DDP.

Construction: The DDP constructor takes a reference to the local
  module, and broadcasts "state_dict()" from the process with rank 0
  to all other processes in the group to make sure that all model
  replicas start from the exact same state. Then, each DDP process
  creates a local "Reducer", which later will take care of the
  gradients synchronization during the backward pass. To improve
  communication efficiency, the "Reducer" organizes parameter
  gradients into buckets, and reduces one bucket at a time. Bucket
  size can be configured by setting the bucket_cap_mb argument in
  DDP constructor. The mapping from parameter gradients to buckets is
  determined at the construction time, based on the bucket size limit
  and parameter sizes. Model parameters are allocated into buckets in
  (roughly) the reverse order of "Model.parameters()" from the given
  model. The reason for using the reverse order is because DDP expects
gradients to become ready during the backward pass in approximately
  that order. The figure below shows an example. Note that, the
  "grad0" and "grad1" are in "bucket1", and the other two gradients
  are in "bucket0". Of course, this assumption might not always be
  true, and when that happens it could hurt DDP backward speed as the
  "Reducer" cannot kick off the communication at the earliest possible
  time. Besides bucketing, the "Reducer" also registers autograd hooks
  during construction, one hook per parameter. These hooks will be
  triggered during the backward pass when the gradient becomes ready.

Forward Pass: The DDP takes the input and passes it to the local
  model, and then analyzes the output from the local model if
  "find_unused_parameters" is set to "True". This mode allows running
  backward on a subgraph of the model, and DDP finds out which
  parameters are involved in the backward pass by traversing the
  autograd graph from the model output and marking all unused
parameters as ready for reduction. During the backward pass, the
  "Reducer" would only wait for unready parameters, but it would still
  reduce all buckets. Marking a parameter gradient as ready does not
  help DDP skip buckets as for now, but it will prevent DDP from
  waiting for absent gradients forever during the backward pass. Note
  that traversing the autograd graph introduces extra overheads, so
  applications should only set "find_unused_parameters" to "True" when
  necessary.

Backward Pass: The "backward()" function is directly invoked on
  the loss "Tensor", which is out of DDP's control, and DDP uses
  autograd hooks registered at construction time to trigger gradients
  synchronizations. When one gradient becomes ready, its corresponding
  DDP hook on that grad accumulator will fire, and DDP will then mark
  that parameter gradient as ready for reduction. When gradients in
  one bucket are all ready, the "Reducer" kicks off an asynchronous
"allreduce" on that bucket to calculate mean of gradients across all
  processes. When all buckets are ready, the "Reducer" will block
  waiting for all "allreduce" operations to finish. When this is done,
  averaged gradients are written to the "param.grad" field of all
  parameters. So after the backward pass, the grad field on the same
  corresponding parameter across different DDP processes should be the
  same.

Optimizer Step: From the optimizer's perspective, it is
  optimizing a local model. Model replicas on all DDP processes can
  keep in sync because they all start from the same state and they
  have the same averaged gradients in every iteration.

[image: ddp_grad_sync.png][image]
Note:
DDP requires "Reducer" instances on all processes to invoke
  "allreduce" in exactly the same order, which is done by always
  running "allreduce" in the bucket index order instead of actual
  bucket ready order. Mismatched "allreduce" order across processes