"torch.utils.data.DataLoader"'s documentation for how to properly set
up random seeds in workers with its "worker_init_fn" option.
My recurrent network doesn't work with data parallelism
There is a subtlety in using the "pack sequence -> recurrent network
-> unpack sequence" pattern in a "Module" with "DataParallel" or
"data_parallel()". Input to each the "forward()" on each device will
only be part of the entire input. Because the unpack operation
"torch.nn.utils.rnn.pad_packed_sequence()" by default only pads up to
the longest input it sees, i.e., the longest on that particular
device, size mismatches will happen when results are gathered
together. Therefore, you can instead take advantage of the
"total_length" argument of "pad_packed_sequence()" to make sure that
the "forward()" calls return sequences of same length. For example,
you can write:
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence


class MyModule(nn.Module):
       # ... init, other methods, etc.
   # padded_input is of shape [B x T x *] (batch_first mode) and contains
   # the sequences sorted by lengths
   #   B is the batch size
   #   T is max sequence length
   def forward(self, padded_input, input_lengths):
       total_length = padded_input.size(1)  # get the max sequence length
       packed_input = pack_padded_sequence(padded_input, input_lengths,
                                           batch_first=True)
       packed_output, _ = self.my_lstm(packed_input)
       output, _ = pad_packed_sequence(packed_output, batch_first=True,
                                       total_length=total_length)
       return output

m = MyModule().cuda()
   dp_m = nn.DataParallel(m)
Additionally, extra care needs to be taken when batch dimension is dim
"1" (i.e., "batch_first=False") with data parallelism. In this case,


the first argument of pack_padded_sequence "padding_input" will be of
shape "[T x B x *]" and should be scattered along dim "1", but the
second argument "input_lengths" will be of shape "[B]" and should be
scattered along dim "0". Extra code to manipulate the tensor shapes
will be needed.


MPS backend
"mps" device enables high-performance training on GPU for MacOS
devices with Metal programming framework.  It introduces a new device
to map Machine Learning computational graphs and primitives on highly
efficient Metal Performance Shaders Graph framework and tuned kernels
provided by Metal Performance Shaders framework respectively.
The new MPS backend extends the PyTorch ecosystem and provides
existing scripts capabilities to setup and run operations on GPU.
To get started, simply move your Tensor and Module to the "mps"
device:
# Check that MPS is available
   if not torch.backends.mps.is_available():
       if not torch.backends.mps.is_built():
           print("MPS not available because the current PyTorch install was not "
                 "built with MPS enabled.")
       else:
           print("MPS not available because the current MacOS version is not 12.3+ "
                 "and/or you do not have an MPS-enabled device on this machine.")
else:


else:
       mps_device = torch.device("mps")
   # Create a Tensor directly on the mps device
   x = torch.ones(5, device=mps_device)
   # Or
   x = torch.ones(5, device="mps")

   # Any operation happens on the GPU
   y = x * 2

   # Move your model to mps just like any other device
   model = YourFavoriteNet()
   model.to(mps_device)

   # Now every call runs on the GPU
   pred = model(x)


Distributed Data Parallel
Warning:
The implementation of "torch.nn.parallel.DistributedDataParallel"
  evolves over time. This design note is written based on the state as
  of v1.4.
"torch.nn.parallel.DistributedDataParallel" (DDP) transparently
performs distributed data parallel training. This page describes how
it works and reveals implementation details.
Example
Let us start with a simple "torch.nn.parallel.DistributedDataParallel"
example. This example uses a "torch.nn.Linear" as the local model,
wraps it with DDP, and then runs one forward pass, one backward pass,
and an optimizer step on the DDP model. After that, parameters on the
local model will be updated, and all models on different processes
should be exactly the same.
import torch
   import torch.distributed as dist
   import torch.multiprocessing as mp
   import torch.nn as nn
   import torch.optim as optim
   from torch.nn.parallel import DistributedDataParallel as DDP


def example(rank, world_size):
       # create default process group
       dist.init_process_group("gloo", rank=rank, world_size=world_size)
       # create local model
       model = nn.Linear(10, 10).to(rank)
       # construct DDP model
       ddp_model = DDP(model, device_ids=[rank])
       # define loss function and optimizer
       loss_fn = nn.MSELoss()
       optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)
   # forward pass
   outputs = ddp_model(torch.randn(20, 10).to(rank))
   labels = torch.randn(20, 10).to(rank)
   # backward pass
   loss_fn(outputs, labels).backward()
   # update parameters
   optimizer.step()

def main():
       world_size = 2
       mp.spawn(example,
           args=(world_size,),
           nprocs=world_size,
           join=True)
if name=="main":
       # Environment variables which need to be
       # set when using c10d's default "env"
       # initialization mode.
       os.environ["MASTER_ADDR"] = "localhost"


os.environ["MASTER_ADDR"] = "localhost"
       os.environ["MASTER_PORT"] = "29500"
       main()
DDP works with TorchDynamo.  When used with TorchDynamo, apply the DDP
model wrapper before compiling the model, such that torchdynamo can
apply "DDPOptimizer" (graph-break optimizations) based on DDP bucket
sizes.  (See TorchDynamo DDPOptimizer for more information.)
TorchDynamo support for DDP currently requires setting
static_graph=False, due to interactions between the graph tracing
process and DDP's mechanism for observing operations happening on its
module, but this should be fixed ultimately.
ddp_model = DDP(model, device_ids=[rank])
   ddp_model = torch.compile(ddp_model)
Internal Design
This section reveals how it works under the hood of
"torch.nn.parallel.DistributedDataParallel" by diving into details of
every step in one iteration.

Prerequisite: DDP relies on c10d "ProcessGroup" for
  communications. Hence, applications must create "ProcessGroup"


instances before constructing DDP.

Construction: The DDP constructor takes a reference to the local
  module, and broadcasts "state_dict()" from the process with rank 0
  to all other processes in the group to make sure that all model
  replicas start from the exact same state. Then, each DDP process
  creates a local "Reducer", which later will take care of the
  gradients synchronization during the backward pass. To improve
  communication efficiency, the "Reducer" organizes parameter
  gradients into buckets, and reduces one bucket at a time. Bucket
  size can be configured by setting the bucket_cap_mb argument in
  DDP constructor. The mapping from parameter gradients to buckets is
  determined at the construction time, based on the bucket size limit
  and parameter sizes. Model parameters are allocated into buckets in
  (roughly) the reverse order of "Model.parameters()" from the given
  model. The reason for using the reverse order is because DDP expects


gradients to become ready during the backward pass in approximately
  that order. The figure below shows an example. Note that, the
  "grad0" and "grad1" are in "bucket1", and the other two gradients
  are in "bucket0". Of course, this assumption might not always be
  true, and when that happens it could hurt DDP backward speed as the
  "Reducer" cannot kick off the communication at the earliest possible
  time. Besides bucketing, the "Reducer" also registers autograd hooks
  during construction, one hook per parameter. These hooks will be
  triggered during the backward pass when the gradient becomes ready.

Forward Pass: The DDP takes the input and passes it to the local
  model, and then analyzes the output from the local model if
  "find_unused_parameters" is set to "True". This mode allows running
  backward on a subgraph of the model, and DDP finds out which
  parameters are involved in the backward pass by traversing the
  autograd graph from the model output and marking all unused


parameters as ready for reduction. During the backward pass, the
  "Reducer" would only wait for unready parameters, but it would still
  reduce all buckets. Marking a parameter gradient as ready does not
  help DDP skip buckets as for now, but it will prevent DDP from
  waiting for absent gradients forever during the backward pass. Note
  that traversing the autograd graph introduces extra overheads, so
  applications should only set "find_unused_parameters" to "True" when
  necessary.

Backward Pass: The "backward()" function is directly invoked on
  the loss "Tensor", which is out of DDP's control, and DDP uses
  autograd hooks registered at construction time to trigger gradients
  synchronizations. When one gradient becomes ready, its corresponding
  DDP hook on that grad accumulator will fire, and DDP will then mark
  that parameter gradient as ready for reduction. When gradients in
  one bucket are all ready, the "Reducer" kicks off an asynchronous


"allreduce" on that bucket to calculate mean of gradients across all
  processes. When all buckets are ready, the "Reducer" will block
  waiting for all "allreduce" operations to finish. When this is done,
  averaged gradients are written to the "param.grad" field of all
  parameters. So after the backward pass, the grad field on the same
  corresponding parameter across different DDP processes should be the
  same.

Optimizer Step: From the optimizer's perspective, it is
  optimizing a local model. Model replicas on all DDP processes can
  keep in sync because they all start from the same state and they
  have the same averaged gradients in every iteration.

[image: ddp_grad_sync.png][image]
Note:
DDP requires "Reducer" instances on all processes to invoke
  "allreduce" in exactly the same order, which is done by always
  running "allreduce" in the bucket index order instead of actual
  bucket ready order. Mismatched "allreduce" order across processes


can lead to wrong results or DDP backward hang.
Implementation
Below are pointers to the DDP implementation components. The stacked
graph shows the structure of the code.
ProcessGroup

ProcessGroup.hpp: contains the abstract API of all process group
  implementations. The "c10d" library provides 3 implementations out
  of the box, namely, ProcessGroupGloo, ProcessGroupNCCL, and
  ProcessGroupMPI. "DistributedDataParallel" uses
  "ProcessGroup::broadcast()" to send model states from the process
  with rank 0 to others during initialization and
  "ProcessGroup::allreduce()" to sum gradients.

Store.hpp: assists the rendezvous service for process group
  instances to find each other.

DistributedDataParallel

distributed.py: is the Python entry point for DDP. It implements the
  initialization steps and the "forward" function for the
  "nn.parallel.DistributedDataParallel" module which call into C++


libraries. Its "_sync_param" function performs intra-process
  parameter synchronization when one DDP process works on multiple
  devices, and it also broadcasts model buffers from the process with
  rank 0 to all other processes. The inter-process parameter
  synchronization happens in "Reducer.cpp".

comm.h: implements the coalesced broadcast helper function which is
  invoked to broadcast model states during initialization and
  synchronize model buffers before the forward pass.

reducer.h: provides the core implementation for gradient
  synchronization in the backward pass. It has three entry point
  functions:

"Reducer": The constructor is called in "distributed.py" which
    registers "Reducer::autograd_hook()" to gradient accumulators.

"autograd_hook()" function will be invoked by the autograd engine
    when a gradient becomes ready.

"prepare_for_backward()" is called at the end of DDP forward pass
    in "distributed.py". It traverses the autograd graph to find


unused parameters when "find_unused_parameters" is set to "True"
    in DDP constructor.
[image: ddp_code.png][image]
TorchDynamo DDPOptimizer
DDP's performance advantage comes from overlapping allreduce
collectives with computations during backwards. AotAutograd prevents
this overlap when used with TorchDynamo for compiling a whole forward
and whole backward graph, because allreduce ops are launched by
autograd hooks after the whole optimized backwards computation
finishes.
TorchDynamo's DDPOptimizer helps by breaking the forward graph at the
logical boundaries of DDP's allreduce buckets during backwards.  Note:
the goal is to break the graph during backwards, and the simplest
implementation is to break the forward graphs and then call
AotAutograd and compilation on each section.  This allows DDP's
allreduce hooks to fire in-between sections of backwards, and schedule
communications to overlap with compute.


communications to overlap with compute.
See this blog post for a more in-depth explanation and experimental
results, or read the docs and code at
torch/_dynamo/optimizations/distributed.py
To Debug DDPOptimizer, set torch._dynamo.config.log_level to DEBUG
(for full graph dumps) or INFO (for basic info about bucket
boundaries).  To disable DDPOptimizer, set
torch._dynamo.config.optimize_ddp=False. DDP and TorchDynamo should
still work correctly without DDPOptimizer, but with performance
degradation.


Features for large-scale deployments
* Fleet-wide operator profiling

API usage logging

Attaching metadata to saved TorchScript models

Build environment considerations

Common extension points

This note talks about several extension points and tricks that might
be useful when running PyTorch within a larger system or operating
multiple systems using PyTorch in a larger organization.
It doesn't cover topics of deploying models to production. Check
"torch.jit" or one of the corresponding tutorials.
The note assumes that you either build PyTorch from source in your
organization or have an ability to statically link additional code to
be loaded when PyTorch is used. Therefore, many of the hooks are
exposed as C++ APIs that can be triggered once in a centralized place,
e.g. in static initialization code.
Fleet-wide operator profiling
PyTorch comes with "torch.autograd.profiler" capable of measuring time


taken by individual operators on demand. One can use the same
mechanism to do "always ON" measurements for any process running
PyTorch. It might be useful for gathering information about PyTorch
workloads running in a given process or across the entire set of
machines.
New callbacks for any operator invocation can be added with
"torch::addGlobalCallback". Hooks will be called with
"torch::RecordFunction" struct that describes invocation context (e.g.
name). If enabled, "RecordFunction::inputs()" contains arguments of
the function represented as "torch::IValue" variant type. Note, that
inputs logging is relatively expensive and thus has to be enabled
explicitly.
The operator callbacks also have access to
"c10::ThreadLocalDebugInfo::get()" interface that returns a pointer to
the struct holding the debug information. This debug information can
be set earlier by using "at::DebugInfoGuard" object. Debug information
is propagated through the forward (including async "fork" tasks) and


backward passes and can be useful for passing some extra information
about execution environment (e.g. model id) from the higher layers of
the application down to the operator callbacks.
Invoking callbacks adds some overhead, so usually it's useful to just
randomly sample operator invocations. This can be enabled on per-
callback basis with an optional sampling rate passed into
"torch::addGlobalCallback".
Note, that "addGlobalCallback" is not thread-safe and can be called
only when no PyTorch operator is running. Usually, it's a good idea to
call them once during initialization.
Here's an example:
// Called somewhere in the program beginning
   void init() {
       // Sample one in a hundred operator runs randomly
       addGlobalCallback(
         RecordFunctionCallback(
           &onFunctionEnter,
           &onFunctionExit)
         .needsInputs(true)
         .samplingProb(0.01)
       );
       // Note, to enable observers in the model calling thread,


// call enableRecordFunction() in the thread before running a model
   }
void onFunctionEnter(const RecordFunction& fn) {
       std::cerr << "Before function " << fn.name()
                 << " with " << fn.inputs().size() << " inputs" << std::endl;
   }
void onFunctionExit(const RecordFunction& fn) {
       std::cerr << "After function " << fn.name();
   }
API usage logging
When running in a broader ecosystem, for example in managed job
scheduler, it's often useful to track which binaries invoke particular
PyTorch APIs. There exists simple instrumentation injected at several
important API points that triggers a given callback. Because usually
PyTorch is invoked in one-off python scripts, the callback fires only
once for a given process for each of the APIs.
"c10::SetAPIUsageHandler" can be used to register API usage
instrumentation handler. Passed argument is going to be an "api key"
identifying used point, for example "python.import" for PyTorch


extension import or "torch.script.compile" if TorchScript compilation
was triggered.
SetAPIUsageLogger( {
       std::cerr << "API was used: " << event_name << std::endl;
   });
Note for developers: new API trigger points can be added in code with
"C10_LOG_API_USAGE_ONCE("my_api")" in C++ or
"torch._C._log_api_usage_once("my.api")" in Python.
Attaching metadata to saved TorchScript models
TorchScript modules can be saved as an archive file that bundles
serialized parameters and module code as TorchScript (see
"torch.jit.save()"). It's often convenient to bundle additional
information together with the model, for example, description of model
producer or auxiliary artifacts.
It can be achieved by passing the "_extra_files" argument to
"torch.jit.save()" and "torch::jit::load" to store and retrieve
arbitrary binary blobs during saving process. Since TorchScript files


are regular ZIP archives, extra information gets stored as regular
files inside archive's "extra/" directory.
There's also a global hook allowing to attach extra files to any
TorchScript archive produced in the current process. It might be
useful to tag models with producer metadata, akin to JPEG metadata
produced by digital cameras. Example usage might look like:
SetExportModuleExtraFilesHook( {
       ExtraFilesMap files;
       files["producer_info.json"] = "{\"user\": \"" + getenv("USER") + "\"}";
       return files;
   });
Build environment considerations
TorchScript's compilation needs to have access to the original python
files as it uses python's "inspect.getsource" call. In certain
production environments it might require explicitly deploying ".py"
files along with precompiled ".pyc".
Common extension points
PyTorch APIs are generally loosely coupled and it's easy to replace a


component with specialized version. Common extension points include:

Custom operators implemented in C++ - see tutorial for more details.

Custom data reading can be often integrated directly by invoking
  corresponding python library. Existing functionality of
  "torch.utils.data" can be utilized by extending "Dataset" or
  "IterableDataset".


Numerical accuracy
In modern computers, floating point numbers are represented using IEEE
754 standard. For more details on floating point arithmetics and IEEE
754 standard, please see Floating point arithmetic In particular, note
that floating point provides limited accuracy (about 7 decimal digits
for single precision floating point numbers, about 16 decimal digits
for double precision floating point numbers) and that floating point
addition and multiplication are not associative, so the order of the
operations affects the results. Because of this, PyTorch is not
guaranteed to produce bitwise identical results for floating point
computations that are mathematically identical. Similarly, bitwise
identical results are not guaranteed across PyTorch releases,
individual commits, or different platforms. In particular, CPU and GPU
results can be different even for bitwise-identical inputs and even
after controlling for the sources of randomness.
Batched computations or slice computations


Batched computations or slice computations
Many operations in PyTorch support batched computation, where the same
operation is performed for the elements of the batches of inputs. An
example of this is "torch.mm()" and "torch.bmm()". It is possible to
implement batched computation as a loop over batch elements, and apply
the necessary math operations to the individual batch elements, for
efficiency reasons we are not doing that, and typically perform
computation for the whole batch. The mathematical libraries that we
are calling, and PyTorch internal implementations of operations can
produces slightly different results in this case, compared to non-
batched computations. In particular, let "A" and "B" be 3D tensors
with the dimensions suitable for batched matrix multiplication. Then
"(A@B)[0]" (the first element of the batched result) is not guaranteed
to be bitwise identical to "A[0]@B[0]" (the matrix product of the


first elements of the input batches) even though mathematically it's
an identical computation.
Similarly, an operation applied to a tensor slice is not guaranteed to
produce results that are identical to the slice of the result of the
same operation applied to the full tensor. E.g. let "A" be a
2-dimensional tensor. "A.sum(-1)[0]" is not guaranteed to be bitwise
equal to "A[:,0].sum()".
Extremal values
When inputs contain large values such that intermediate results may
overflow the range of the used datatype, the end result may overflow
too, even though it is representable in the original datatype. E.g.:
import torch
   a=torch.tensor([1e20, 1e20]) # fp32 type by default
   a.norm() # produces tensor(inf)
   a.double().norm() # produces tensor(1.4142e+20, dtype=torch.float64), representable in fp32
Linear algebra ("torch.linalg")
Non-finite values
The external libraries (backends) that "torch.linalg" uses provide no


guarantees on their behaviour when the inputs have non-finite values
like "inf" or "NaN". As such, neither does PyTorch. The operations may
return a tensor with non-finite values, or raise an exception, or even
segfault.
Consider using "torch.isfinite()" before calling these functions to
detect this situation.
Extremal values in linalg
Functions within "torch.linalg" have more Extremal Values than other
PyTorch functions.
Solvers and Inverses assume that the input matrix "A" is invertible.
If it is close to being non-invertible (for example, if it has a very
small singular value), then these algorithms may silently return
incorrect results. These matrices are said to be ill-conditioned. If
provided with ill-conditioned inputs, the result of these functions
they may vary when using the same inputs on different devices or when
using different backends via the keyword "driver".
Spectral operations like "svd", "eig", and "eigh" may also return


incorrect results (and their gradients may be infinite) when their
inputs have singular values that are close to each other. This is
because the algorithms used to compute these decompositions struggle
to converge for these inputs.
Running the computation in "float64" (as NumPy does by default) often
helps, but it does not solve these issues in all cases. Analyzing the
spectrum of the inputs via "torch.linalg.svdvals()" or their condition
number via "torch.linalg.cond()" may help to detect these issues.
TensorFloat-32(TF32) on Nvidia Ampere devices
On Ampere Nvidia GPUs, PyTorch can use TensorFloat32 (TF32) to speed
up mathematically intensive operations, in particular matrix
multiplications and convolutions. When an operation is performed using
TF32 tensor cores, only the first 10 bits of the input mantissa are
read. This may reduce accuracy and produce surprising results (e.g.,
multiplying a matrix by the identity matrix may produce results that


are different from the input). By default, TF32 tensor cores are
disabled for matrix multiplications and enabled for convolutions,
although most neural network workloads have the same convergence
behavior when using TF32 as they have with fp32. We recommend enabling
TF32 tensor cores for matrix multiplications with
"torch.backends.cuda.matmul.allow_tf32 = True" if your network does
not need full float32 precision. If your network needs full float32
precision for both matrix multiplications and convolutions, then TF32
tensor cores can also be disabled for convolutions with
"torch.backends.cudnn.allow_tf32 = False".
For more information see TensorFloat32.
Reduced Precision Reduction for FP16  and BF16 GEMMs
Half-precision GEMM operations are typically done with intermediate
accumulations (reduction) in single-precision for numerical accuracy
and improved resilience to overflow. For performance, certain GPU


architectures, especially more recent ones, allow a few truncations of
the intermediate accumulation results to the reduced precision (e.g.,
half-precision). This change is often benign from the perspective of
model convergence, though it may lead to unexpected results (e.g.,
"inf" values when the final result should be be representable in half-
precision). If reduced-precision reductions are problematic, they can
be turned off with
"torch.backends.cuda.matmul.allow_fp16_reduced_precision_reduction =
False"
A similar flag exists for BF16 GEMM operations and is turned off by
default. If BF16 reduced-precision reductions are problematic, they
can be turned off with
"torch.backends.cuda.matmul.allow_bf16_reduced_precision_reduction =
False"
For more information see allow_fp16_reduced_precision_reduction and
allow_bf16_reduced_precision_reduction
Reduced Precision FP16 and BF16 GEMMs and Convolutions on AMD Instinct MI200 devices


====================================================================================
On AMD Instinct MI200 GPUs, the FP16 and BF16 V_DOT2 and MFMA matrix
instructions flush input and output denormal values to zero. FP32 and
FP64 MFMA matrix instructions do not flush input and output denormal
values to zero. The affected instructions are only used by rocBLAS
(GEMM) and MIOpen (convolution) kernels; all other PyTorch operations
will not encounter this behavior. All other supported AMD GPUs will
not encounter this behavior.
rocBLAS and MIOpen provide alternate implementations for affected FP16
operations. Alternate implementations for BF16 operations are not
provided; BF16 numbers have a larger dynamic range than FP16 numbers
and are less likely to encounter denormal values. For the FP16
alternate implementations, FP16 input values are cast to an
intermediate BF16 value and then cast back to FP16 output after the
accumulate FP32 operations. In this way, the input and output types
are unchanged.


are unchanged.
When training using FP16 precision, some models may fail to converge
with FP16 denorms flushed to zero. Denormal values more frequently
occur in the backward pass of training during gradient calculation.
PyTorch by default will use the rocBLAS and MIOpen alternate
implementations during the backward pass. The default behavior can be
overridden using environment variables, ROCBLAS_INTERNAL_FP16_ALT_IMPL
and MIOPEN_DEBUG_CONVOLUTION_ATTRIB_FP16_ALT_IMPL. The behavior of
these environment variables is as follows:
+-----------------+-------------+-------------+
|                 | forward     | backward    |
|=================|=============|=============|
| Env unset       | original    | alternate   |
+-----------------+-------------+-------------+
| Env set to 1    | alternate   | alternate   |
+-----------------+-------------+-------------+
| Env set to 0    | original    | original    |
+-----------------+-------------+-------------+


+-----------------+-------------+-------------+
The following is the list of operations where rocBLAS may be used:

torch.addbmm

torch.addmm

torch.baddbmm

torch.bmm

torch.mm

torch.nn.GRUCell

torch.nn.LSTMCell

torch.nn.Linear

torch.sparse.addmm

the following torch._C._ConvBackend implementations:

slowNd

slowNd_transposed

slowNd_dilated

slowNd_dilated_transposed

The following is the list of operations where MIOpen may be used:

torch.nn.Conv[Transpose]Nd

the following torch._C._ConvBackend implementations:

ConvBackend::Miopen

ConvBackend::MiopenDepthwise

ConvBackend::MiopenTranspose


Broadcasting semantics
Many PyTorch operations support NumPy's broadcasting semantics. See
https://numpy.org/doc/stable/user/basics.broadcasting.html for
details.
In short, if a PyTorch operation supports broadcast, then its Tensor
arguments can be automatically expanded to be of equal sizes (without
making copies of the data).
General semantics
Two tensors are "broadcastable" if the following rules hold:

Each tensor has at least one dimension.

When iterating over the dimension sizes, starting at the trailing
  dimension, the dimension sizes must either be equal, one of them is
  1, or one of them does not exist.

For Example:

x=torch.empty(5,7,3)
y=torch.empty(5,7,3)
   # same shapes are always broadcastable (i.e. the above rules always hold)
x=torch.empty((0,))
y=torch.empty(2,2)
   # x and y are not broadcastable, because x does not have at least 1 dimension

# can line up trailing dimensions

x=torch.empty(5,3,4,1)


x=torch.empty(5,3,4,1)
y=torch.empty(  3,1,1)
   # x and y are broadcastable.
   # 1st trailing dimension: both have size 1
   # 2nd trailing dimension: y has size 1
   # 3rd trailing dimension: x size == y size
   # 4th trailing dimension: y dimension doesn't exist

# but:

x=torch.empty(5,2,4,1)
y=torch.empty(  3,1,1)
   # x and y are not broadcastable, because in the 3rd trailing dimension 2 != 3

If two tensors "x", "y" are "broadcastable", the resulting tensor size
is calculated as follows:

If the number of dimensions of "x" and "y" are not equal, prepend 1
  to the dimensions of the tensor with fewer dimensions to make them
  equal length.

Then, for each dimension size, the resulting dimension size is the
  max of the sizes of "x" and "y" along that dimension.

For Example:
# can line up trailing dimensions to make reading easier

x=torch.empty(5,1,4,1)
y=torch.empty(  3,1,1)
(x+y).size()
   torch.Size([5, 3, 4, 1])

# but not necessary:


but not necessary:

x=torch.empty(1)
y=torch.empty(3,1,7)
(x+y).size()
   torch.Size([3, 1, 7])
x=torch.empty(5,2,4,1)
y=torch.empty(3,1,1)
(x+y).size()
   RuntimeError: The size of tensor a (2) must match the size of tensor b (3) at non-singleton dimension 1

In-place semantics
One complication is that in-place operations do not allow the in-place
tensor to change shape as a result of the broadcast.
For Example:

x=torch.empty(5,3,4,1)
y=torch.empty(3,1,1)
(x.add_(y)).size()
   torch.Size([5, 3, 4, 1])

# but:

x=torch.empty(1,3,1)
y=torch.empty(3,1,7)
(x.add_(y)).size()
   RuntimeError: The expanded size of the tensor (1) must match the existing size (7) at non-singleton dimension 2.

Backwards compatibility
Prior versions of PyTorch allowed certain pointwise functions to
execute on tensors with different shapes, as long as the number of


elements in each tensor was equal.  The pointwise operation would then
be carried out by viewing each tensor as 1-dimensional.  PyTorch now
supports broadcasting and the "1-dimensional" pointwise behavior is
considered deprecated and will generate a Python warning in cases
where tensors are not broadcastable, but have the same number of
elements.
Note that the introduction of broadcasting can cause backwards
incompatible changes in the case where two tensors do not have the
same shape, but are broadcastable and have the same number of
elements. For Example:

torch.add(torch.ones(4,1), torch.randn(4))

would previously produce a Tensor with size: torch.Size([4,1]), but
now produces a Tensor with size: torch.Size([4,4]). In order to help
identify cases in your code where backwards incompatibilities
introduced by broadcasting may exist, you may set
torch.utils.backcompat.broadcast_warning.enabled to True, which
will generate a python warning in such cases.
For Example:


For Example:

torch.utils.backcompat.broadcast_warning.enabled=True
torch.add(torch.ones(4,1), torch.ones(4))
   main:1: UserWarning: self and other do not have the same shape, but are broadcastable, and have the same number of elements.
   Changing behavior in a backwards incompatible manner to broadcasting rather than viewing as 1-dimensional.


HIP (ROCm) semantics
ROCmâ¢ is AMDâs open source software platform for GPU-accelerated high
performance computing and machine learning. HIP is ROCm's C++ dialect
designed to ease conversion of CUDA applications to portable C++ code.
HIP is used when converting existing CUDA applications like PyTorch to
portable C++ and for new projects that require portability between AMD
and NVIDIA.
HIP Interfaces Reuse the CUDA Interfaces
PyTorch for HIP intentionally reuses the existing "torch.cuda"
interfaces. This helps to accelerate the porting of existing PyTorch
code and models because very few code changes are necessary, if any.
The example from CUDA semantics will work exactly the same for HIP:
cuda = torch.device('cuda')     # Default HIP device
   cuda0 = torch.device('cuda:0')  # 'rocm' or 'hip' are not valid, use 'cuda'
   cuda2 = torch.device('cuda:2')  # GPU 2 (these are 0-indexed)
x = torch.tensor([1., 2.], device=cuda0)


x = torch.tensor([1., 2.], device=cuda0)
   # x.device is device(type='cuda', index=0)
   y = torch.tensor([1., 2.]).cuda()
   # y.device is device(type='cuda', index=0)
with torch.cuda.device(1):
       # allocates a tensor on GPU 1
       a = torch.tensor([1., 2.], device=cuda)
   # transfers a tensor from CPU to GPU 1
   b = torch.tensor([1., 2.]).cuda()
   # a.device and b.device are device(type='cuda', index=1)

   # You can also use ``Tensor.to`` to transfer a tensor:
   b2 = torch.tensor([1., 2.]).to(device=cuda)
   # b.device and b2.device are device(type='cuda', index=1)

   c = a + b
   # c.device is device(type='cuda', index=1)

   z = x + y
   # z.device is device(type='cuda', index=0)

   # even within a context, you can specify the device
   # (or give a GPU index to the .cuda call)
   d = torch.randn(2, device=cuda2)
   e = torch.randn(2).to(cuda2)
   f = torch.randn(2).cuda(cuda2)


f = torch.randn(2).cuda(cuda2)
       # d.device, e.device, and f.device are all device(type='cuda', index=2)
Checking for HIP
Whether you are using PyTorch for CUDA or HIP, the result of calling
"is_available()" will be the same. If you are using a PyTorch that has
been built with GPU support, it will return True. If you must check
which version of PyTorch you are using, refer to this example below:
if torch.cuda.is_available() and torch.version.hip:
       # do something specific for HIP
   elif torch.cuda.is_available() and torch.version.cuda:
       # do something specific for CUDA
TensorFloat-32(TF32) on ROCm
TF32 is not supported on ROCm.
Memory management
PyTorch uses a caching memory allocator to speed up memory
allocations. This allows fast memory deallocation without device
synchronizations. However, the unused memory managed by the allocator
will still show as if used in "rocm-smi". You can use


"memory_allocated()" and "max_memory_allocated()" to monitor memory
occupied by tensors, and use "memory_reserved()" and
"max_memory_reserved()" to monitor the total amount of memory managed
by the caching allocator. Calling "empty_cache()" releases all
unused cached memory from PyTorch so that those can be used by
other GPU applications. However, the occupied GPU memory by tensors
will not be freed so it can not increase the amount of GPU memory
available for PyTorch.
For more advanced users, we offer more comprehensive memory
benchmarking via "memory_stats()". We also offer the capability to
capture a complete snapshot of the memory allocator state via
"memory_snapshot()", which can help you understand the underlying
allocation patterns produced by your code.
To debug memory errors, set "PYTORCH_NO_CUDA_MEMORY_CACHING=1" in your
environment to disable caching.
hipFFT/rocFFT plan cache
Setting the size of the cache for hipFFT/rocFFT plans is not
supported.


supported.
torch.distributed backends
Currently, only the "nccl" and "gloo" backends for torch.distributed
are supported on ROCm.
CUDA API to HIP API mappings in C++
Please refer: https://rocmdocs.amd.com/en/latest/Programming_Guides/H
IP_API_Guide.html
NOTE: The CUDA_VERSION macro, cudaRuntimeGetVersion and
cudaDriverGetVersion APIs do not semantically map to the same values
as HIP_VERSION macro, hipRuntimeGetVersion and hipDriverGetVersion
APIs. Please do not use them interchangeably when doing version
checks.
For example: Instead of using
"#if defined(CUDA_VERSION) && CUDA_VERSION >= 11000" to implicitly
exclude ROCm/HIP,
use the following to not take the code path for ROCm/HIP:
"#if defined(CUDA_VERSION) && CUDA_VERSION >= 11000 &&
!defined(USE_ROCM)"
Alternatively, if it is desired to take the code path for ROCm/HIP:
"#if (defined(CUDA_VERSION) && CUDA_VERSION >= 11000) ||
defined(USE_ROCM)"


defined(USE_ROCM)"
Or if it is desired to take the code path for ROCm/HIP only for
specific HIP versions:
"#if (defined(CUDA_VERSION) && CUDA_VERSION >= 11000) ||
(defined(USE_ROCM) && ROCM_VERSION >= 40300)"
Refer to CUDA Semantics doc
For any sections not listed here, please refer to the CUDA semantics
doc: CUDA semantics
Enabling kernel asserts
Kernel asserts are supported on ROCm, but they are disabled due to
performance overhead. It can be enabled by recompiling the PyTorch
from source.
Please add below line as an argument to cmake command parameters:
-DROCM_FORCE_ENABLE_GPU_ASSERTS:BOOL=ON


Generic Join Context Manager
The generic join context manager facilitates distributed training on
uneven inputs. This page outlines the API of the relevant classes:
"Join", "Joinable", and "JoinHook". For a tutorial, see Distributed
Training with Uneven Inputs Using the Join Context Manager.
class torch.distributed.algorithms.Join(joinables, enable=True, throw_on_early_termination=False, **kwargs)
This class defines the generic join context manager, which allows
   custom hooks to be called after a process joins. These hooks should
   shadow the collective communications of non-joined processes to
   prevent hanging and erroring and to ensure algorithmic correctness.
   Refer to "JoinHook" for details about the hook definition.
Warning:
 The context manager requires each participating "Joinable" to
 call the method "notify_join_context()" before its own per-
 iteration collective communications to ensure correctness.

Warning:


Warning:
 The context manager requires that all "process_group" attributes
 in the "JoinHook" objects are the same. If there are multiple
 "JoinHook" objects, then the "device" of the first is used. The
 process group and device information is used for checking for
 non- joined processes and for notifying processes to throw an
 exception if "throw_on_early_termination" is enabled, both of
 which using an all- reduce.

Parameters:
      * joinables (List[Joinable]) -- a list of the
        participating "Joinable" s; their hooks are iterated over in
        the given order.
  * **enable** (*bool*) -- a flag enabling uneven input detection;
    setting to "False" disables the context manager's
    functionality and should only be set when the user knows the
    inputs will not be uneven (default: "True").

  * **throw_on_early_termination** (*bool*) -- a flag controlling
    whether to throw an exception upon detecting uneven inputs


(default: "False").
Example:
  >>> import os
  >>> import torch
  >>> import torch.distributed as dist
  >>> import torch.multiprocessing as mp
  >>> import torch.nn.parallel.DistributedDataParallel as DDP
  >>> import torch.distributed.optim.ZeroRedundancyOptimizer as ZeRO
  >>> from torch.distributed.algorithms.join import Join
  >>>
  >>> # On each spawned worker
  >>> def worker(rank):
  >>>     dist.init_process_group("nccl", rank=rank, world_size=2)
  >>>     model = DDP(torch.nn.Linear(1, 1).to(rank), device_ids=[rank])
  >>>     optim = ZeRO(model.parameters(), torch.optim.Adam, lr=0.01)
  >>>     # Rank 1 gets one more input than rank 0
  >>>     inputs = [torch.tensor([1.]).to(rank) for _ in range(10 + rank)]
  >>>     with Join([model, optim]):
  >>>         for input in inputs:
  >>>             loss = model(input).sum()
  >>>             loss.backward()
  >>>             optim.step()


        optim.step()
  >>>     # All ranks reach here without hanging/erroring

static notify_join_context(joinable)
  Notifies the join context manager that the calling process has
  not yet joined; then, if "throw_on_early_termination=True",
  checks if uneven inputs have been detected (i.e. if one process
  has already joined) and throws an exception if so.

  This method should be called from a "Joinable" object before its
  per-iteration collective communications. For example, this
  should be called at the beginning of the forward pass in
  "DistributedDataParallel".

  Only the first "Joinable" object passed into the context manager
  performs the collective communications in this method, and for
  the others, this method is vacuous.

  Parameters:
     **joinable** (*Joinable*) -- the "Joinable" object calling
     this method.

  Returns:
     An async work handle for the all-reduce meant to notify the


context manager that the process has not yet joined if
         "joinable" is the first one passed into the context manager;
         "None" otherwise.
class torch.distributed.algorithms.Joinable
This defines an abstract base class for joinable classes. A
   joinable class (inheriting from "Joinable") should implement
   "join_hook()", which returns a "JoinHook" instance, in addition to
   "join_device()" and "join_process_group()" that return device and
   process group information, respectively.
abstract property join_device: device
  Returns the device from which to perform collective
  communications needed by the join context manager implementation
  itself.

abstract join_hook(**kwargs)
  Returns a "JoinHook" instance for the given "Joinable".

  Parameters:
     **kwargs** (*dict*) -- a "dict" containing any keyword
     arguments to modify the behavior of the join hook at run
     time; all "Joinable" instances sharing the same join context


manager are forwarded the same value for "kwargs".
  Return type:
     *JoinHook*

abstract property join_process_group: Any
  Returns the process group for the collective communications
  needed by the join context manager itself.

class torch.distributed.algorithms.JoinHook
This defines a join hook, which provides two entry points in the
   join context manager: a main hook, which is called repeatedly while
   there exists a non-joined process, and a post-hook, which is called
   once all processes have joined.
To implement a join hook for the generic join context manager,
   define a class that inherits from "JoinHook" and override
   "main_hook()" and "post_hook()" as appropriate.
main_hook()
  This hook is called repeatedly while there exists a non-joined
  process to shadow collective communications in one training
  iteration (i.e. in one forward pass, backward pass, and
  optimizer step).

post_hook(is_last_joiner)


post_hook(is_last_joiner)
  This hook is called after all processes have joined. It is
  passed an additional "bool" argument "is_last_joiner", which
  indicates if the rank is one of the last to join.

  Parameters:
     **is_last_joiner** (*bool*) -- "True" if the rank is one of
     the last to join; "False" otherwise.


torch.utils.tensorboard
Before going further, more details on TensorBoard can be found at
https://www.tensorflow.org/tensorboard/
Once you've installed TensorBoard, these utilities let you log PyTorch
models and metrics into a directory for visualization within the
TensorBoard UI. Scalars, images, histograms, graphs, and embedding
visualizations are all supported for PyTorch models and tensors as
well as Caffe2 nets and blobs.
The SummaryWriter class is your main entry to log data for consumption
and visualization by TensorBoard. For example:
import torch
   import torchvision
   from torch.utils.tensorboard import SummaryWriter
   from torchvision import datasets, transforms
# Writer will output to ./runs/ directory by default
   writer = SummaryWriter()
transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (0.5,))])
   trainset = datasets.MNIST('mnist_train', train=True, download=True, transform=transform)


trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)
   model = torchvision.models.resnet50(False)
   # Have ResNet model take in grayscale rather than RGB
   model.conv1 = torch.nn.Conv2d(1, 64, kernel_size=7, stride=2, padding=3, bias=False)
   images, labels = next(iter(trainloader))
grid = torchvision.utils.make_grid(images)
   writer.add_image('images', grid, 0)
   writer.add_graph(model, images)
   writer.close()
This can then be visualized with TensorBoard, which should be
installable and runnable with:
pip install tensorboard
   tensorboard --logdir=runs
Lots of information can be logged for one experiment. To avoid
cluttering the UI and have better result clustering, we can group
plots by naming them hierarchically. For example, "Loss/train" and
"Loss/test" will be grouped together, while "Accuracy/train" and
"Accuracy/test" will be grouped separately in the TensorBoard
interface.
from torch.utils.tensorboard import SummaryWriter
   import numpy as np


import numpy as np
writer = SummaryWriter()
for n_iter in range(100):
       writer.add_scalar('Loss/train', np.random.random(), n_iter)
       writer.add_scalar('Loss/test', np.random.random(), n_iter)
       writer.add_scalar('Accuracy/train', np.random.random(), n_iter)
       writer.add_scalar('Accuracy/test', np.random.random(), n_iter)
Expected result:
[image]
class torch.utils.tensorboard.writer.SummaryWriter(log_dir=None, comment='', purge_step=None, max_queue=10, flush_secs=120, filename_suffix='')
Writes entries directly to event files in the log_dir to be
   consumed by TensorBoard.
The SummaryWriter class provides a high-level API to create an
   event file in a given directory and add summaries and events to it.
   The class updates the file contents asynchronously. This allows a
   training program to call methods to add data to the file directly
   from the training loop, without slowing down training.


init(log_dir=None, comment='', purge_step=None, max_queue=10, flush_secs=120, filename_suffix='')
  Creates a *SummaryWriter* that will write out events and
  summaries to the event file.

  Parameters:
     * **log_dir** (*str*) -- Save directory location. Default is
       runs/**CURRENT_DATETIME_HOSTNAME**, which changes after
       each run. Use hierarchical folder structure to compare
       between runs easily. e.g. pass in 'runs/exp1', 'runs/exp2',
       etc. for each new experiment to compare across them.

     * **comment** (*str*) -- Comment log_dir suffix appended to
       the default "log_dir". If "log_dir" is assigned, this
       argument has no effect.

     * **purge_step** (*int*) -- When logging crashes at step T+X
       and restarts at step T, any events whose global_step larger
       or equal to T will be purged and hidden from TensorBoard.
       Note that crashed and resumed experiments should have the


same "log_dir".
     * **max_queue** (*int*) -- Size of the queue for pending
       events and summaries before one of the 'add' calls forces a
       flush to disk. Default is ten items.

     * **flush_secs** (*int*) -- How often, in seconds, to flush
       the pending events and summaries to disk. Default is every
       two minutes.

     * **filename_suffix** (*str*) -- Suffix added to all event
       filenames in the log_dir directory. More details on
       filename construction in tensorboard.summary.writer.event_
       file_writer.EventFileWriter.

  Examples:

     from torch.utils.tensorboard import SummaryWriter

     # create a summary writer with automatically generated folder name.
     writer = SummaryWriter()
     # folder location: runs/May04_22-14-54_s-MacBook-Pro.local/

     # create a summary writer using the specified folder name.
     writer = SummaryWriter("my_experiment")


writer = SummaryWriter("my_experiment")
         # folder location: my_experiment
     # create a summary writer with comment appended.
     writer = SummaryWriter(comment="LR_0.1_BATCH_16")
     # folder location: runs/May04_22-14-54_s-MacBook-Pro.localLR_0.1_BATCH_16/

add_scalar(tag, scalar_value, global_step=None, walltime=None, new_style=False, double_precision=False)
  Add scalar data to summary.

  Parameters:
     * **tag** (*str*) -- Data identifier

     * **scalar_value** (*float** or **string/blobname*) -- Value
       to save

     * **global_step** (*int*) -- Global step value to record

     * **walltime** (*float*) -- Optional override default
       walltime (time.time()) with seconds after epoch of event

     * **new_style** (*boolean*) -- Whether to use new style
       (tensor field) or old style (simple_value field). New style
       could lead to faster data loading.

  Examples:


Examples:
     from torch.utils.tensorboard import SummaryWriter
     writer = SummaryWriter()
     x = range(100)
     for i in x:
         writer.add_scalar('y=2x', i * 2, i)
     writer.close()

  Expected result:

  [image]

add_scalars(main_tag, tag_scalar_dict, global_step=None, walltime=None)
  Adds many scalar data to summary.

  Parameters:
     * **main_tag** (*str*) -- The parent name for the tags

     * **tag_scalar_dict** (*dict*) -- Key-value pair storing the
       tag and corresponding values

     * **global_step** (*int*) -- Global step value to record

     * **walltime** (*float*) -- Optional override default
       walltime (time.time()) seconds after epoch of event

  Examples:

     from torch.utils.tensorboard import SummaryWriter
     writer = SummaryWriter()
     r = 5
     for i in range(100):
         writer.add_scalars('run_14h', {'xsinx':i*np.sin(i/r),


'xcosx':i*np.cos(i/r),
                                             'tanx': np.tan(i/r)}, i)
         writer.close()
         # This call adds three values to the same scalar plot with the tag
         # 'run_14h' in TensorBoard's scalar section.
  Expected result:

  [image]

add_histogram(tag, values, global_step=None, bins='tensorflow', walltime=None, max_bins=None)
  Add histogram to summary.

  Parameters:
     * **tag** (*str*) -- Data identifier

     * **values** (*torch.Tensor**, **numpy.ndarray**, or
       **string/blobname*) -- Values to build histogram

     * **global_step** (*int*) -- Global step value to record

     * **bins** (*str*) -- One of {'tensorflow','auto', 'fd',
       ...}. This determines how the bins are made. You can find
       other options in: https://docs.scipy.org/doc/numpy/referen
       ce/generated/numpy.histogram.html


ce/generated/numpy.histogram.html
     * **walltime** (*float*) -- Optional override default
       walltime (time.time()) seconds after epoch of event

  Examples:

     from torch.utils.tensorboard import SummaryWriter
     import numpy as np
     writer = SummaryWriter()
     for i in range(10):
         x = np.random.random(1000)
         writer.add_histogram('distribution centers', x + i, i)
     writer.close()

  Expected result:

  [image]

add_image(tag, img_tensor, global_step=None, walltime=None, dataformats='CHW')
  Add image data to summary.

  Note that this requires the "pillow" package.

  Parameters:
     * **tag** (*str*) -- Data identifier

     * **img_tensor** (*torch.Tensor**, **numpy.ndarray**, or
       **string/blobname*) -- Image data

     * **global_step** (*int*) -- Global step value to record

     * **walltime** (*float*) -- Optional override default


walltime (time.time()) seconds after epoch of event
     * **dataformats** (*str*) -- Image data format specification
       of the form CHW, HWC, HW, WH, etc.

  Shape:
     img_tensor: Default is (3, H, W). You can use
     "torchvision.utils.make_grid()" to convert a batch of tensor
     into 3xHxW format or call "add_images" and let us do the job.
     Tensor with (1, H, W), (H, W), (H, W, 3) is also suitable as
     long as corresponding "dataformats" argument is passed, e.g.
     "CHW", "HWC", "HW".

  Examples:

     from torch.utils.tensorboard import SummaryWriter
     import numpy as np
     img = np.zeros((3, 100, 100))
     img[0] = np.arange(0, 10000).reshape(100, 100) / 10000
     img[1] = 1 - np.arange(0, 10000).reshape(100, 100) / 10000

     img_HWC = np.zeros((100, 100, 3))
     img_HWC[:, :, 0] = np.arange(0, 10000).reshape(100, 100) / 10000


img_HWC[:, :, 1] = 1 - np.arange(0, 10000).reshape(100, 100) / 10000
     writer = SummaryWriter()
     writer.add_image('my_image', img, 0)

     # If you have non-default dimension setting, set the dataformats argument.
     writer.add_image('my_image_HWC', img_HWC, 0, dataformats='HWC')
     writer.close()

  Expected result:

  [image]

add_images(tag, img_tensor, global_step=None, walltime=None, dataformats='NCHW')
  Add batched image data to summary.

  Note that this requires the "pillow" package.

  Parameters:
     * **tag** (*str*) -- Data identifier

     * **img_tensor** (*torch.Tensor**, **numpy.ndarray**, or
       **string/blobname*) -- Image data

     * **global_step** (*int*) -- Global step value to record

     * **walltime** (*float*) -- Optional override default
       walltime (time.time()) seconds after epoch of event

     * **dataformats** (*str*) -- Image data format specification


of the form NCHW, NHWC, CHW, HWC, HW, WH, etc.
  Shape:
     img_tensor: Default is (N, 3, H, W). If "dataformats" is
     specified, other shape will be accepted. e.g. NCHW or NHWC.

  Examples:

     from torch.utils.tensorboard import SummaryWriter
     import numpy as np

     img_batch = np.zeros((16, 3, 100, 100))
     for i in range(16):
         img_batch[i, 0] = np.arange(0, 10000).reshape(100, 100) / 10000 / 16 * i
         img_batch[i, 1] = (1 - np.arange(0, 10000).reshape(100, 100) / 10000) / 16 * i

     writer = SummaryWriter()
     writer.add_images('my_image_batch', img_batch, 0)
     writer.close()

  Expected result:

  [image]

add_figure(tag, figure, global_step=None, close=True, walltime=None)
  Render matplotlib figure into an image and add it to summary.

  Note that this requires the "matplotlib" package.

  Parameters:
     * **tag** (*str*) -- Data identifier


tag (str) -- Data identifier * **figure** (*matplotlib.pyplot.figure*) -- Figure or a list
   of figures

 * **global_step** (*int*) -- Global step value to record

 * **close** (*bool*) -- Flag to automatically close the
   figure

 * **walltime** (*float*) -- Optional override default
   walltime (time.time()) seconds after epoch of event

add_video(tag, vid_tensor, global_step=None, fps=4, walltime=None)
  Add video data to summary.

  Note that this requires the "moviepy" package.

  Parameters:
     * **tag** (*str*) -- Data identifier

     * **vid_tensor** (*torch.Tensor*) -- Video data

     * **global_step** (*int*) -- Global step value to record

     * **fps** (*float** or **int*) -- Frames per second

     * **walltime** (*float*) -- Optional override default
       walltime (time.time()) seconds after epoch of event

  Shape:


Shape:
         vid_tensor: (N, T, C, H, W). The values should lie in [0,
         255] for type uint8 or [0, 1] for type float.
add_audio(tag, snd_tensor, global_step=None, sample_rate=44100, walltime=None)
  Add audio data to summary.

  Parameters:
     * **tag** (*str*) -- Data identifier

     * **snd_tensor** (*torch.Tensor*) -- Sound data

     * **global_step** (*int*) -- Global step value to record

     * **sample_rate** (*int*) -- sample rate in Hz

     * **walltime** (*float*) -- Optional override default
       walltime (time.time()) seconds after epoch of event

  Shape:
     snd_tensor: (1, L). The values should lie between [-1, 1].

add_text(tag, text_string, global_step=None, walltime=None)
  Add text data to summary.

  Parameters:
     * **tag** (*str*) -- Data identifier

     * **text_string** (*str*) -- String to save

     * **global_step** (*int*) -- Global step value to record


walltime (float) -- Optional override default
           walltime (time.time()) seconds after epoch of event
Examples:
 writer.add_text('lstm', 'This is an lstm', 0)
 writer.add_text('rnn', 'This is an rnn', 10)

add_graph(model, input_to_model=None, verbose=False, use_strict_trace=True)
  Add graph data to summary.

  Parameters:
     * **model** (*torch.nn.Module*) -- Model to draw.

     * **input_to_model** (*torch.Tensor** or **list of
       torch.Tensor*) -- A variable or a tuple of variables to be
       fed.

     * **verbose** (*bool*) -- Whether to print graph structure in
       console.

     * **use_strict_trace** (*bool*) -- Whether to pass keyword
       argument *strict* to *torch.jit.trace*. Pass False when you
       want the tracer to record your mutable container types
       (list, dict)

add_embedding(mat, metadata=None, label_img=None, global_step=None, tag='default', metadata_header=None)


Add embedding projector data to summary.
  Parameters:
     * **mat** (*torch.Tensor** or **numpy.ndarray*) -- A matrix
       which each row is the feature vector of the data point

     * **metadata** (*list*) -- A list of labels, each element
       will be convert to string

     * **label_img** (*torch.Tensor*) -- Images correspond to each
       data point

     * **global_step** (*int*) -- Global step value to record

     * **tag** (*str*) -- Name for the embedding

  Shape:
     mat: (N, D), where N is number of data and D is feature
     dimension

     label_img: (N, C, H, W)

  Examples:

     import keyword
     import torch
     meta = []
     while len(meta)<100:
         meta = meta+keyword.kwlist # get some strings
     meta = meta[:100]

     for i, v in enumerate(meta):
         meta[i] = v+str(i)

     label_img = torch.rand(100, 3, 10, 32)
     for i in range(100):


for i in range(100):
             label_img[i]*=i/100.0
     writer.add_embedding(torch.randn(100, 5), metadata=meta, label_img=label_img)
     writer.add_embedding(torch.randn(100, 5), label_img=label_img)
     writer.add_embedding(torch.randn(100, 5), metadata=meta)

add_pr_curve(tag, labels, predictions, global_step=None, num_thresholds=127, weights=None, walltime=None)
  Adds precision recall curve. Plotting a precision-recall curve
  lets you understand your model's performance under different
  threshold settings. With this function, you provide the ground
  truth labeling (T/F) and prediction confidence (usually the
  output of your model) for each target. The TensorBoard UI will
  let you choose the threshold interactively.

  Parameters:
     * **tag** (*str*) -- Data identifier

     * **labels** (*torch.Tensor**, **numpy.ndarray**, or
       **string/blobname*) -- Ground truth data. Binary label for
       each element.


each element.
     * **predictions** (*torch.Tensor**, **numpy.ndarray**, or
       **string/blobname*) -- The probability that an element be
       classified as true. Value should be in [0, 1]

     * **global_step** (*int*) -- Global step value to record

     * **num_thresholds** (*int*) -- Number of thresholds used to
       draw the curve.

     * **walltime** (*float*) -- Optional override default
       walltime (time.time()) seconds after epoch of event

  Examples:

     from torch.utils.tensorboard import SummaryWriter
     import numpy as np
     labels = np.random.randint(2, size=100)  # binary label
     predictions = np.random.rand(100)
     writer = SummaryWriter()
     writer.add_pr_curve('pr_curve', labels, predictions, 0)
     writer.close()

add_custom_scalars(layout)
  Create special chart by collecting charts tags in 'scalars'.
  Note that this function can only be called once for each


SummaryWriter() object. Because it only provides metadata to
      tensorboard, the function can be called before or after the
      training loop.
  Parameters:
     **layout** (*dict*) -- {categoryName: *charts*}, where
     *charts* is also a dictionary {chartName:
     *ListOfProperties*}. The first element in *ListOfProperties*
     is the chart's type (one of **Multiline** or **Margin**) and
     the second element should be a list containing the tags you
     have used in add_scalar function, which will be collected
     into the new chart.

  Examples:

     layout = {'Taiwan':{'twse':['Multiline',['twse/0050', 'twse/2330']]},
                  'USA':{ 'dow':['Margin',   ['dow/aaa', 'dow/bbb', 'dow/ccc']],
                       'nasdaq':['Margin',   ['nasdaq/aaa', 'nasdaq/bbb', 'nasdaq/ccc']]}}

     writer.add_custom_scalars(layout)

add_mesh(tag, vertices, colors=None, faces=None, config_dict=None, global_step=None, walltime=None)


Add meshes or 3D point clouds to TensorBoard. The visualization
      is based on Three.js, so it allows users to interact with the
      rendered object. Besides the basic definitions such as vertices,
      faces, users can further provide camera parameter, lighting
      condition, etc. Please check https://threejs.org/docs/index.htm
      l#manual/en/introduction/Creating-a-scene for advanced usage.
  Parameters:
     * **tag** (*str*) -- Data identifier

     * **vertices** (*torch.Tensor*) -- List of the 3D coordinates
       of vertices.

     * **colors** (*torch.Tensor*) -- Colors for each vertex

     * **faces** (*torch.Tensor*) -- Indices of vertices within
       each triangle. (Optional)

     * **config_dict** -- Dictionary with ThreeJS classes names
       and configuration.

     * **global_step** (*int*) -- Global step value to record

     * **walltime** (*float*) -- Optional override default


walltime (time.time()) seconds after epoch of event
  Shape:
     vertices: (B, N, 3). (batch, number_of_vertices, channels)

     colors: (B, N, 3). The values should lie in [0, 255] for type
     *uint8* or [0, 1] for type *float*.

     faces: (B, N, 3). The values should lie in [0,
     number_of_vertices] for type *uint8*.

  Examples:

     from torch.utils.tensorboard import SummaryWriter
     vertices_tensor = torch.as_tensor([
         [1, 1, 1],
         [-1, -1, 1],
         [1, -1, -1],
         [-1, 1, -1],
     ], dtype=torch.float).unsqueeze(0)
     colors_tensor = torch.as_tensor([
         [255, 0, 0],
         [0, 255, 0],
         [0, 0, 255],
         [255, 0, 255],
     ], dtype=torch.int).unsqueeze(0)
     faces_tensor = torch.as_tensor([
         [0, 2, 3],
         [0, 3, 1],
         [0, 1, 2],
         [1, 3, 2],
     ], dtype=torch.int).unsqueeze(0)


], dtype=torch.int).unsqueeze(0)
     writer = SummaryWriter()
     writer.add_mesh('my_mesh', vertices=vertices_tensor, colors=colors_tensor, faces=faces_tensor)

     writer.close()

add_hparams(hparam_dict, metric_dict, hparam_domain_discrete=None, run_name=None)
  Add a set of hyperparameters to be compared in TensorBoard.

  Parameters:
     * **hparam_dict** (*dict*) -- Each key-value pair in the
       dictionary is the name of the hyper parameter and it's
       corresponding value. The type of the value can be one of
       *bool*, *string*, *float*, *int*, or *None*.

     * **metric_dict** (*dict*) -- Each key-value pair in the
       dictionary is the name of the metric and it's corresponding
       value. Note that the key used here should be unique in the
       tensorboard record. Otherwise the value you added by
       "add_scalar" will be displayed in hparam plugin. In most
       cases, this is unwanted.


cases, this is unwanted.
     * **hparam_domain_discrete** -- (Optional[Dict[str,
       List[Any]]]) A dictionary that contains names of the
       hyperparameters and all discrete values they can hold

     * **run_name** (*str*) -- Name of the run, to be included as
       part of the logdir. If unspecified, will use current
       timestamp.

  Examples:

     from torch.utils.tensorboard import SummaryWriter
     with SummaryWriter() as w:
         for i in range(5):
             w.add_hparams({'lr': 0.1*i, 'bsize': i},
                           {'hparam/accuracy': 10*i, 'hparam/loss': 10*i})

  Expected result:

  [image]

flush()
  Flushes the event file to disk. Call this method to make sure
  that all pending events have been written to disk.

close()