shrinath-suresh/blogs-docs-splitted huggingface dataset:
unscaled  automatically during "step()".
  Simple example, using "unscale_()" to enable clipping of
  unscaled gradients:

     ...
     scaler.scale(loss).backward()
     scaler.unscale_(optimizer)
     torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm)
     scaler.step(optimizer)
     scaler.update()

  Parameters:
     **optimizer** (*torch.optim.Optimizer*) -- Optimizer that
     owns the gradients to be unscaled.

  Note:

    "unscale_()" does not incur a CPU-GPU sync.

  Warning:

    "unscale_()" should only be called once per optimizer per
    "step()" call, and only after all gradients for that
    optimizer's assigned parameters have been accumulated. Calling
    "unscale_()" twice for a given optimizer between each "step()"
    triggers a RuntimeError.

  Warning:

    "unscale_()" may unscale sparse gradients out of place,
    replacing the ".grad" attribute.

update(new_scale=None)
update(new_scale=None)
  Updates the scale factor.

  If any optimizer steps were skipped the scale is multiplied by
  "backoff_factor" to reduce it. If "growth_interval" unskipped
  iterations occurred consecutively, the scale is multiplied by
  "growth_factor" to increase it.

  Passing "new_scale" sets the new scale value manually.
  ("new_scale" is not used directly, it's used to fill
  GradScaler's internal scale tensor. So if "new_scale" was a
  tensor, later in-place changes to that tensor will not further
  affect the scale GradScaler uses internally.)

  Parameters:
     **new_scale** (float or "torch.cuda.FloatTensor", optional,
     default=None) -- New scale factor.

  Warning:

    "update()" should only be called at the end of the iteration,
    after "scaler.step(optimizer)" has been invoked for all
    optimizers used this iteration.

Autocast Op Reference
Op Eligibility
Op Eligibility
Ops that run in "float64" or non-floating-point dtypes are not
eligible, and will run in these types whether or not autocast is
enabled.
Only out-of-place ops and Tensor methods are eligible. In-place
variants and calls that explicitly supply an "out=..." Tensor are
allowed in autocast-enabled regions, but won't go through autocasting.
For example, in an autocast-enabled region "a.addmm(b, c)" can
autocast, but "a.addmm_(b, c)" and "a.addmm(b, c, out=d)" cannot. For
best performance and stability, prefer out-of-place ops in autocast-
enabled regions.
Ops called with an explicit "dtype=..." argument are not eligible, and
will produce output that respects the "dtype" argument.
CUDA Op-Specific Behavior
The following lists describe the behavior of eligible ops in autocast-
enabled regions. These ops always go through autocasting whether they
are invoked as part of a "torch.nn.Module", as a function, or as a
"torch.Tensor" method. If functions are exposed in multiple
namespaces, they go through autocasting regardless of the namespace.
Ops not listed below do not go through autocasting.  They run in the
type defined by their inputs.  However, autocasting may still change
the type in which unlisted ops run if they're downstream from
autocasted ops.
If an op is unlisted, we assume it's numerically stable in "float16".
If you believe an unlisted op is numerically unstable in "float16",
please file an issue.
CUDA Ops that can autocast to "float16"

"__matmul__", "addbmm", "addmm", "addmv", "addr", "baddbmm", "bmm",
"chain_matmul", "multi_dot", "conv1d", "conv2d", "conv3d",
"conv_transpose1d", "conv_transpose2d", "conv_transpose3d", "GRUCell",
"linear", "LSTMCell", "matmul", "mm", "mv", "prelu", "RNNCell"


CUDA Ops that can autocast to "float32"

"pow", "rdiv", "rpow", "rtruediv", "acos", "asin",
"binary_cross_entropy_with_logits", "cosh", "cosine_embedding_loss",
"cdist", "cosine_similarity", "cross_entropy", "cumprod", "cumsum",
"dist", "erfinv", "exp", "expm1", "group_norm",
"hinge_embedding_loss", "kl_div", "l1_loss", "layer_norm", "log",
"log_softmax", "log10", "log1p", "log2", "margin_ranking_loss",
"mse_loss", "multilabel_margin_loss", "multi_margin_loss", "nll_loss",
"norm", "normalize", "pdist", "poisson_nll_loss", "pow", "prod",
"reciprocal", "rsqrt", "sinh", "smooth_l1_loss", "soft_margin_loss",
"softmax", "softmin", "softplus", "sum", "renorm", "tan",
"triplet_margin_loss"
CUDA Ops that promote to the widest input type
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
These ops don't require a particular dtype for stability, but take
multiple inputs and require that the inputs' dtypes match.  If all of
the inputs are "float16", the op runs in "float16".  If any of the
inputs is "float32", autocast casts all inputs to "float32" and runs
the op in "float32".
the op in "float32".
"addcdiv", "addcmul", "atan2", "bilinear", "cross", "dot",
"grid_sample", "index_put", "scatter_add", "tensordot"
Some ops not listed here (e.g., binary ops like "add") natively
promote inputs without autocasting's intervention.  If inputs are a
mixture of "float16" and "float32", these ops run in "float32" and
produce "float32" output, regardless of whether autocast is enabled.
Prefer "binary_cross_entropy_with_logits" over "binary_cross_entropy"
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The backward passes of "torch.nn.functional.binary_cross_entropy()"
(and "torch.nn.BCELoss", which wraps it) can produce gradients that
aren't representable in "float16".  In autocast-enabled regions, the
forward input may be "float16", which means the backward gradient must
be representable in "float16" (autocasting "float16" forward inputs to
"float32" doesn't help, because that cast must be reversed in
backward). Therefore, "binary_cross_entropy" and "BCELoss" raise an
error in autocast-enabled regions.
Many models use a sigmoid layer right before the binary cross entropy
layer. In this case, combine the two layers using
"torch.nn.functional.binary_cross_entropy_with_logits()" or
"torch.nn.BCEWithLogitsLoss".  "binary_cross_entropy_with_logits" and
"BCEWithLogits" are safe to autocast.
CPU Op-Specific Behavior
The following lists describe the behavior of eligible ops in autocast-
enabled regions. These ops always go through autocasting whether they
are invoked as part of a "torch.nn.Module", as a function, or as a
"torch.Tensor" method. If functions are exposed in multiple
namespaces, they go through autocasting regardless of the namespace.
Ops not listed below do not go through autocasting.  They run in the
type defined by their inputs.  However, autocasting may still change
the type in which unlisted ops run if they're downstream from
autocasted ops.
If an op is unlisted, we assume it's numerically stable in "bfloat16".
If you believe an unlisted op is numerically unstable in "bfloat16",
please file an issue.
CPU Ops that can autocast to "bfloat16"
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
"conv1d", "conv2d", "conv3d", "bmm", "mm", "baddbmm", "addmm",
"addbmm", "linear", "matmul", "_convolution"
CPU Ops that can autocast to "float32"
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
"conv_transpose1d", "conv_transpose2d", "conv_transpose3d",
"avg_pool3d", "binary_cross_entropy", "grid_sampler",
"grid_sampler_2d", "_grid_sampler_2d_cpu_fallback", "grid_sampler_3d",
"polar", "prod", "quantile", "nanquantile", "stft", "cdist", "trace",
"view_as_complex", "cholesky", "cholesky_inverse", "cholesky_solve",
"inverse", "lu_solve", "orgqr", "inverse", "ormqr", "pinverse",
"max_pool3d", "max_unpool2d", "max_unpool3d", "adaptive_avg_pool3d",
"reflection_pad1d", "reflection_pad2d", "replication_pad1d",
"replication_pad2d", "replication_pad3d", "mse_loss", "ctc_loss",
"kl_div", "multilabel_margin_loss", "fft_fft", "fft_ifft", "fft_fft2",
"fft_ifft2", "fft_fftn", "fft_ifftn", "fft_rfft", "fft_irfft",
"fft_rfft2", "fft_irfft2", "fft_rfftn", "fft_irfftn", "fft_hfft",
"fft_ihfft", "linalg_matrix_norm", "linalg_cond",
"linalg_matrix_rank", "linalg_solve", "linalg_cholesky",
"linalg_svdvals", "linalg_eigvals", "linalg_eigvalsh", "linalg_inv",
"linalg_householder_product", "linalg_tensorinv",
"linalg_tensorsolve", "fake_quantize_per_tensor_affine", "eig",
"geqrf", "lstsq", "_lu_with_info", "qr", "solve", "svd", "symeig",
"triangular_solve", "fractional_max_pool2d", "fractional_max_pool3d",
"adaptive_max_pool3d", "multilabel_margin_loss_forward", "linalg_qr",
"linalg_cholesky_ex", "linalg_svd", "linalg_eig", "linalg_eigh",
"linalg_lstsq", "linalg_inv_ex"
CPU Ops that promote to the widest input type
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
These ops don't require a particular dtype for stability, but take
multiple inputs and require that the inputs' dtypes match.  If all of
the inputs are "bfloat16", the op runs in "bfloat16".  If any of the
inputs is "float32", autocast casts all inputs to "float32" and runs
the op in "float32".
"cat", "stack", "index_copy"
Some ops not listed here (e.g., binary ops like "add") natively
promote inputs without autocasting's intervention.  If inputs are a
mixture of "bfloat16" and "float32", these ops run in "float32" and
produce "float32" output, regardless of whether autocast is enabled.
torch._dynamo
Warning:
This module is an early prototype and is subject to change.
torch._dynamo.allow_in_graph(fn)
Customize which functions TorchDynamo will include in the generated
   graph. Similar to torch.fx.wrap().
  torch._dynamo.allow_in_graph(my_custom_function)

  @torch._dynamo.optimize(...)
  def fn(a):
      x = torch.add(x, 1)
      x = my_custom_function(x)
      x = torch.add(x, 1)
      return x

  fn(...)

Will capture a single graph containing my_custom_function().
torch._dynamo.disallow_in_graph(fn)
Customize which functions TorchDynamo will exclude in the generated
   graph and force a graph break on.
  torch._dynamo.disallow_in_graph(torch.sub)

  @torch._dynamo.optimize(...)
  def fn(a):
      x = torch.add(x, 1)
      x = torch.sub(x, 1)
      x = torch.add(x, 1)
      return x

  fn(...)

Will break the graph on torch.sub, and give two graphs each with
a single torch.add() op.
torch._dynamo.graph_break()
Force a graph break
torch._dynamo.optimize(backend='inductor', *, nopython=False, guard_export_fn=None, guard_fail_fn=None, disable=False, dynamic=False)
The main entrypoint of TorchDynamo.  Do graph capture and call
   backend() to optimize extracted graphs.
Parameters:
      * backend -- One of the two things: - Either, a
        function/callable taking a torch.fx.GraphModule and
        example_inputs and returning a python callable that runs the
        graph faster. One can also provide additional context for the
        backend, like torch.jit.fuser("fuser2"), by setting the
        backend_ctx_ctor attribute. See
        AOTAutogradMemoryEfficientFusionWithContext for the usage. -
        Or, a string backend name in torch._dynamo.list_backends()
  * **nopython** -- If True, graph breaks will be errors and there
    will be a single whole-program graph.
will be a single whole-program graph.
  * **disable** -- If True, turn this decorator into a no-op

  * **dynamic** -- If True, turn on dynamic shapes support

Example Usage:
  @torch._dynamo.optimize()
  def toy_example(a, b):
      ...

torch._dynamo.optimize_assert(backend, *, hooks=Hooks(guard_export_fn=None, guard_fail_fn=None), export=False, dynamic=False)
The same as torch._dynamo.optimize(backend, nopython=True)
torch._dynamo.run(fn=None)
Don't do any dynamic compiles, just use prior optimizations
torch._dynamo.disable(fn=None)
Decorator and context manager to disable TorchDynamo
torch._dynamo.reset()
Clear all compile caches and restore initial state
torch._dynamo.list_backends()
Return valid strings that can be passed to:
  @torch._dynamo.optimize(<backend>)
  def foo(...):
     ....

torch._dynamo.skip(fn=None)
Skip frames associated with the function code, but still process
   recursively invoked frames
Tensor Views
PyTorch allows a tensor to be a "View" of an existing tensor. View
tensor shares the same underlying data with its base tensor.
Supporting "View" avoids explicit data copy, thus allows us to do fast
and memory efficient reshaping, slicing and element-wise operations.
For example, to get a view of an existing tensor "t", you can call
"t.view(...)".



t = torch.rand(4, 4)
b = t.view(2, 8)
t.storage().data_ptr() == b.storage().data_ptr()  # t and b share the same underlying data.
   True
   # Modifying view tensor changes base tensor as well.
b[0][0] = 3.14
t[0][0]
   tensor(3.14)



Since views share underlying data with its base tensor, if you edit
the data in the view, it will be reflected in the base tensor as well.
Typically a PyTorch op returns a new tensor as output, e.g. "add()".
But in case of view ops, outputs are views of input tensors to avoid
unnecessary data copy. No data movement occurs when creating a view,
view tensor just changes the way it interprets the same data. Taking a
view of contiguous tensor could potentially produce a non-contiguous
tensor. Users should pay additional attention as contiguity might have
implicit performance impact. "transpose()" is a common example.



base = torch.tensor([[0, 1],[2, 3]])
base.is_contiguous()
   True
t = base.transpose(0, 1)  # t is a view of base. No data movement happened here.
   # View tensors might be non-contiguous.
t.is_contiguous()
   False
   # To get a contiguous tensor, call .contiguous() to enforce
   # copying data when t is not contiguous.
c = t.contiguous()



For reference, hereâs a full list of view ops in PyTorch:


Basic slicing and indexing op, e.g. "tensor[0, 2:, 1:7:2]" returns a
  view of base "tensor", see note below.


"adjoint()"


"as_strided()"


"detach()"


"diagonal()"


"expand()"


"expand_as()"


"movedim()"


"narrow()"


"permute()"


"select()"


"squeeze()"


"transpose()"
"select()"


"squeeze()"


"transpose()"


"t()"


"T"


"H"


"mT"


"mH"


"real"


"imag"


"view_as_real()"


"unflatten()"


"unfold()"


"unsqueeze()"


"view()"


"view_as()"


"unbind()"


"split()"


"hsplit()"


"vsplit()"


"tensor_split()"


"split_with_sizes()"


"swapaxes()"


"swapdims()"


"chunk()"


"indices()" (sparse tensor only)


"values()"  (sparse tensor only)


Note:
When accessing the contents of a tensor via indexing, PyTorch
  follows Numpy behaviors that basic indexing returns views, while
  advanced indexing returns a copy. Assignment via either basic or
  advanced indexing is in-place. See more examples in Numpy indexing
  documentation.
It's also worth mentioning a few ops with special behaviors:


"reshape()", "reshape_as()" and "flatten()" can return either a view
  or new tensor, user code shouldn't rely on whether it's view or not.


"contiguous()" returns itself if input tensor is already
torch.Storage
"torch.Storage" is an alias for the storage class that corresponds
with the default data type ("torch.get_default_dtype()"). For
instance, if the default data type is "torch.float", "torch.Storage"
resolves to "torch.FloatStorage".
The "torch.Storage" and "torch.cuda.Storage" classes, like
"torch.FloatStorage", "torch.IntStorage", etc., are not actually ever
instantiated. Calling their constructors creates a
"torch.TypedStorage" with the appropriate "torch.dtype" and
"torch.device".  "torch.Storage" classes have all of the same
class methods that "torch.TypedStorage" has.
A "torch.TypedStorage" is a contiguous, one-dimensional array of
elements of a particular "torch.dtype". It can be given any
"torch.dtype", and the internal data will be interpreted
appropriately. "torch.TypedStorage" contains a "torch.UntypedStorage"
which holds the data as an untyped array of bytes.
Every strided "torch.Tensor" contains a "torch.TypedStorage", which
stores all of the data that the "torch.Tensor" views.
Warning:
All storage classes except for "torch.UntypedStorage" will be
  removed in the future, and "torch.UntypedStorage" will be used in
  all cases.
class torch.TypedStorage(*args, wrap_storage=None, dtype=None, device=None, _internal=False)
bfloat16()
  Casts this storage to bfloat16 type

bool()
  Casts this storage to bool type

byte()
  Casts this storage to byte type

char()
  Casts this storage to char type

clone()
  Returns a copy of this storage

complex_double()
  Casts this storage to complex double type

complex_float()
  Casts this storage to complex float type

copy_(source, non_blocking=None)
cpu()
  Returns a CPU copy of this storage if it's not already on the
  CPU

cuda(device=None, non_blocking=False, **kwargs)
  Returns a copy of this object in CUDA memory.

  If this object is already in CUDA memory and on the correct
device, then no copy is performed and the original object is
      returned.
  Parameters:
     * **device** (*int*) -- The destination GPU id. Defaults to
       the current device.

     * **non_blocking** (*bool*) -- If "True" and the source is in
       pinned memory, the copy will be asynchronous with respect
       to the host. Otherwise, the argument has no effect.

     * ****kwargs** -- For compatibility, may contain the key
       "async" in place of the "non_blocking" argument.

  Return type:
     *T*

data_ptr()
property device
double()
  Casts this storage to double type

dtype: dtype
element_size()
fill_(value)
float()
  Casts this storage to float type

classmethod from_buffer(args, *kwargs)
classmethod from_file(filename, shared=False, size=0) -> Storage
  If *shared* is *True*, then memory is shared between all
  processes. All changes are written to the file. If *shared* is
False, then the changes on the storage do not affect the file.
  *size* is the number of elements in the storage. If *shared* is
  *False*, then the file must contain at least *size *
  sizeof(Type)* bytes (*Type* is the type of storage). If *shared*
  is *True* the file will be created if needed.

  Parameters:
     * **filename** (*str*) -- file name to map

     * **shared** (*bool*) -- whether to share memory

     * **size** (*int*) -- number of elements in the storage

get_device()
  Return type:
     int

half()
  Casts this storage to half type

int()
  Casts this storage to int type

property is_cuda
is_pinned()
is_shared()
is_sparse = False
long()
  Casts this storage to long type

nbytes()
pickle_storage_type()
pin_memory()
  Coppies the  storage to pinned memory, if it's not already
  pinned.

resize_(size)
share_memory_()
  Moves the storage to shared memory.
Moves the storage to shared memory.
  This is a no-op for storages already in shared memory and for
  CUDA storages, which do not need to be moved for sharing across
  processes. Storages in shared memory cannot be resized.

  Returns: self

short()
  Casts this storage to short type

size()
tolist()
  Returns a list containing the elements of this storage

type(dtype=None, non_blocking=False)
  Returns the type if *dtype* is not provided, else casts this
  object to the specified type.

  If this is already of the correct type, no copy is performed and
  the original object is returned.

  Parameters:
     * **dtype** (*type** or **string*) -- The desired type

     * **non_blocking** (*bool*) -- If "True", and the source is
       in pinned memory and destination is on the GPU or vice
       versa, the copy is performed asynchronously with respect to
       the host. Otherwise, the argument has no effect.
**kwargs -- For compatibility, may contain the key
           "async" in place of the "non_blocking" argument. The
           "async" arg is deprecated.
Return type:
     Union[T, str]


untyped()
  Returns the internal "torch.UntypedStorage"

class torch.UntypedStorage(args, *kwargs)
bfloat16()
  Casts this storage to bfloat16 type

bool()
  Casts this storage to bool type

byte()
  Casts this storage to byte type

char()
  Casts this storage to char type

clone()
  Returns a copy of this storage

complex_double()
  Casts this storage to complex double type

complex_float()
  Casts this storage to complex float type

copy_()
cpu()
  Returns a CPU copy of this storage if it's not already on the
  CPU

cuda(device=None, non_blocking=False, **kwargs)
  Returns a copy of this object in CUDA memory.

  If this object is already in CUDA memory and on the correct
device, then no copy is performed and the original object is
      returned.
  Parameters:
     * **device** (*int*) -- The destination GPU id. Defaults to
       the current device.

     * **non_blocking** (*bool*) -- If "True" and the source is in
       pinned memory, the copy will be asynchronous with respect
       to the host. Otherwise, the argument has no effect.

     * ****kwargs** -- For compatibility, may contain the key
       "async" in place of the "non_blocking" argument.

data_ptr()
device: device
double()
  Casts this storage to double type

element_size()
fill_()
float()
  Casts this storage to float type

static from_buffer()
static from_file(filename, shared=False, size=0) -> Storage
  If *shared* is *True*, then memory is shared between all
  processes. All changes are written to the file. If *shared* is
  *False*, then the changes on the storage do not affect the file.
size is the number of elements in the storage. If shared is
      False, then the file must contain at least size *
      sizeof(Type) bytes (Type is the type of storage). If shared
      is True the file will be created if needed.
  Parameters:
     * **filename** (*str*) -- file name to map

     * **shared** (*bool*) -- whether to share memory

     * **size** (*int*) -- number of elements in the storage

get_device()
  Return type:
     int

half()
  Casts this storage to half type

int()
  Casts this storage to int type

property is_cuda
is_pinned()
is_shared()
is_sparse: bool = False
is_sparse_csr: bool = False
long()
  Casts this storage to long type

mps()
  Returns a CPU copy of this storage if it's not already on the
  CPU

nbytes()
new()
pin_memory()
  Copies the storage to pinned memory, if it's not already pinned.

resize_()
share_memory_()
resize_()
share_memory_()
  Moves the storage to shared memory.

  This is a no-op for storages already in shared memory and for
  CUDA storages, which do not need to be moved for sharing across
  processes. Storages in shared memory cannot be resized.

  Returns: self

short()
  Casts this storage to short type

size()
  Return type:
     int

tolist()
  Returns a list containing the elements of this storage

type(dtype=None, non_blocking=False, **kwargs)
  Returns the type if *dtype* is not provided, else casts this
  object to the specified type.

  If this is already of the correct type, no copy is performed and
  the original object is returned.

  Parameters:
     * **dtype** (*type** or **string*) -- The desired type

     * **non_blocking** (*bool*) -- If "True", and the source is
       in pinned memory and destination is on the GPU or vice
versa, the copy is performed asynchronously with respect to
           the host. Otherwise, the argument has no effect.
     * ****kwargs** -- For compatibility, may contain the key
       "async" in place of the "non_blocking" argument. The
       "async" arg is deprecated.

untyped()
class torch.DoubleStorage(*args, wrap_storage=None, dtype=None, device=None, _internal=False)
dtype: dtype = torch.float64
class torch.FloatStorage(*args, wrap_storage=None, dtype=None, device=None, _internal=False)
dtype: dtype = torch.float32
class torch.HalfStorage(*args, wrap_storage=None, dtype=None, device=None, _internal=False)
dtype: dtype = torch.float16
class torch.LongStorage(*args, wrap_storage=None, dtype=None, device=None, _internal=False)
dtype: dtype = torch.int64
class torch.IntStorage(*args, wrap_storage=None, dtype=None, device=None, _internal=False)
dtype: dtype = torch.int32