shrinath-suresh/blogs-docs-splitted huggingface dataset:
f = torch.randn(2).cuda(cuda2)
       # d.device, e.device, and f.device are all device(type='cuda', index=2)
Checking for HIP
Whether you are using PyTorch for CUDA or HIP, the result of calling
"is_available()" will be the same. If you are using a PyTorch that has
been built with GPU support, it will return True. If you must check
which version of PyTorch you are using, refer to this example below:
if torch.cuda.is_available() and torch.version.hip:
       # do something specific for HIP
   elif torch.cuda.is_available() and torch.version.cuda:
       # do something specific for CUDA
TensorFloat-32(TF32) on ROCm
TF32 is not supported on ROCm.
Memory management
PyTorch uses a caching memory allocator to speed up memory
allocations. This allows fast memory deallocation without device
synchronizations. However, the unused memory managed by the allocator
will still show as if used in "rocm-smi". You can use
"memory_allocated()" and "max_memory_allocated()" to monitor memory
occupied by tensors, and use "memory_reserved()" and
"max_memory_reserved()" to monitor the total amount of memory managed
by the caching allocator. Calling "empty_cache()" releases all
unused cached memory from PyTorch so that those can be used by
other GPU applications. However, the occupied GPU memory by tensors
will not be freed so it can not increase the amount of GPU memory
available for PyTorch.
For more advanced users, we offer more comprehensive memory
benchmarking via "memory_stats()". We also offer the capability to
capture a complete snapshot of the memory allocator state via
"memory_snapshot()", which can help you understand the underlying
allocation patterns produced by your code.
To debug memory errors, set "PYTORCH_NO_CUDA_MEMORY_CACHING=1" in your
environment to disable caching.
hipFFT/rocFFT plan cache
Setting the size of the cache for hipFFT/rocFFT plans is not
supported.
supported.
torch.distributed backends
Currently, only the "nccl" and "gloo" backends for torch.distributed
are supported on ROCm.
CUDA API to HIP API mappings in C++
Please refer: https://rocmdocs.amd.com/en/latest/Programming_Guides/H
IP_API_Guide.html
NOTE: The CUDA_VERSION macro, cudaRuntimeGetVersion and
cudaDriverGetVersion APIs do not semantically map to the same values
as HIP_VERSION macro, hipRuntimeGetVersion and hipDriverGetVersion
APIs. Please do not use them interchangeably when doing version
checks.
For example: Instead of using
"#if defined(CUDA_VERSION) && CUDA_VERSION >= 11000" to implicitly
exclude ROCm/HIP,
use the following to not take the code path for ROCm/HIP:
"#if defined(CUDA_VERSION) && CUDA_VERSION >= 11000 &&
!defined(USE_ROCM)"
Alternatively, if it is desired to take the code path for ROCm/HIP:
"#if (defined(CUDA_VERSION) && CUDA_VERSION >= 11000) ||
defined(USE_ROCM)"
defined(USE_ROCM)"
Or if it is desired to take the code path for ROCm/HIP only for
specific HIP versions:
"#if (defined(CUDA_VERSION) && CUDA_VERSION >= 11000) ||
(defined(USE_ROCM) && ROCM_VERSION >= 40300)"
Refer to CUDA Semantics doc
For any sections not listed here, please refer to the CUDA semantics
doc: CUDA semantics
Enabling kernel asserts
Kernel asserts are supported on ROCm, but they are disabled due to
performance overhead. It can be enabled by recompiling the PyTorch
from source.
Please add below line as an argument to cmake command parameters:
-DROCM_FORCE_ENABLE_GPU_ASSERTS:BOOL=ON
Generic Join Context Manager
The generic join context manager facilitates distributed training on
uneven inputs. This page outlines the API of the relevant classes:
"Join", "Joinable", and "JoinHook". For a tutorial, see Distributed
Training with Uneven Inputs Using the Join Context Manager.
class torch.distributed.algorithms.Join(joinables, enable=True, throw_on_early_termination=False, **kwargs)
This class defines the generic join context manager, which allows
   custom hooks to be called after a process joins. These hooks should
   shadow the collective communications of non-joined processes to
   prevent hanging and erroring and to ensure algorithmic correctness.
   Refer to "JoinHook" for details about the hook definition.
Warning:
 The context manager requires each participating "Joinable" to
 call the method "notify_join_context()" before its own per-
 iteration collective communications to ensure correctness.

Warning:
Warning:
 The context manager requires that all "process_group" attributes
 in the "JoinHook" objects are the same. If there are multiple
 "JoinHook" objects, then the "device" of the first is used. The
 process group and device information is used for checking for
 non- joined processes and for notifying processes to throw an
 exception if "throw_on_early_termination" is enabled, both of
 which using an all- reduce.

Parameters:
      * joinables (List[Joinable]) -- a list of the
        participating "Joinable" s; their hooks are iterated over in
        the given order.
  * **enable** (*bool*) -- a flag enabling uneven input detection;
    setting to "False" disables the context manager's
    functionality and should only be set when the user knows the
    inputs will not be uneven (default: "True").

  * **throw_on_early_termination** (*bool*) -- a flag controlling
    whether to throw an exception upon detecting uneven inputs
(default: "False").
Example:
  >>> import os
  >>> import torch
  >>> import torch.distributed as dist
  >>> import torch.multiprocessing as mp
  >>> import torch.nn.parallel.DistributedDataParallel as DDP
  >>> import torch.distributed.optim.ZeroRedundancyOptimizer as ZeRO
  >>> from torch.distributed.algorithms.join import Join
  >>>
  >>> # On each spawned worker
  >>> def worker(rank):
  >>>     dist.init_process_group("nccl", rank=rank, world_size=2)
  >>>     model = DDP(torch.nn.Linear(1, 1).to(rank), device_ids=[rank])
  >>>     optim = ZeRO(model.parameters(), torch.optim.Adam, lr=0.01)
  >>>     # Rank 1 gets one more input than rank 0
  >>>     inputs = [torch.tensor([1.]).to(rank) for _ in range(10 + rank)]
  >>>     with Join([model, optim]):
  >>>         for input in inputs:
  >>>             loss = model(input).sum()
  >>>             loss.backward()
  >>>             optim.step()
optim.step()
  >>>     # All ranks reach here without hanging/erroring




static notify_join_context(joinable)
  Notifies the join context manager that the calling process has
  not yet joined; then, if "throw_on_early_termination=True",
  checks if uneven inputs have been detected (i.e. if one process
  has already joined) and throws an exception if so.

  This method should be called from a "Joinable" object before its
  per-iteration collective communications. For example, this
  should be called at the beginning of the forward pass in
  "DistributedDataParallel".

  Only the first "Joinable" object passed into the context manager
  performs the collective communications in this method, and for
  the others, this method is vacuous.

  Parameters:
     **joinable** (*Joinable*) -- the "Joinable" object calling
     this method.

  Returns:
     An async work handle for the all-reduce meant to notify the
context manager that the process has not yet joined if
         "joinable" is the first one passed into the context manager;
         "None" otherwise.
class torch.distributed.algorithms.Joinable
This defines an abstract base class for joinable classes. A
   joinable class (inheriting from "Joinable") should implement
   "join_hook()", which returns a "JoinHook" instance, in addition to
   "join_device()" and "join_process_group()" that return device and
   process group information, respectively.
abstract property join_device: device
  Returns the device from which to perform collective
  communications needed by the join context manager implementation
  itself.

abstract join_hook(**kwargs)
  Returns a "JoinHook" instance for the given "Joinable".

  Parameters:
     **kwargs** (*dict*) -- a "dict" containing any keyword
     arguments to modify the behavior of the join hook at run
     time; all "Joinable" instances sharing the same join context
manager are forwarded the same value for "kwargs".
  Return type:
     *JoinHook*

abstract property join_process_group: Any
  Returns the process group for the collective communications
  needed by the join context manager itself.

class torch.distributed.algorithms.JoinHook
This defines a join hook, which provides two entry points in the
   join context manager: a main hook, which is called repeatedly while
   there exists a non-joined process, and a post-hook, which is called
   once all processes have joined.
To implement a join hook for the generic join context manager,
   define a class that inherits from "JoinHook" and override
   "main_hook()" and "post_hook()" as appropriate.
main_hook()
  This hook is called repeatedly while there exists a non-joined
  process to shadow collective communications in one training
  iteration (i.e. in one forward pass, backward pass, and
  optimizer step).

post_hook(is_last_joiner)
torch.utils.tensorboard
Before going further, more details on TensorBoard can be found at
https://www.tensorflow.org/tensorboard/
Once you've installed TensorBoard, these utilities let you log PyTorch
models and metrics into a directory for visualization within the
TensorBoard UI. Scalars, images, histograms, graphs, and embedding
visualizations are all supported for PyTorch models and tensors as
well as Caffe2 nets and blobs.
The SummaryWriter class is your main entry to log data for consumption
and visualization by TensorBoard. For example:
import torch
   import torchvision
   from torch.utils.tensorboard import SummaryWriter
   from torchvision import datasets, transforms
# Writer will output to ./runs/ directory by default
   writer = SummaryWriter()
transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (0.5,))])
   trainset = datasets.MNIST('mnist_train', train=True, download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)
   model = torchvision.models.resnet50(False)
   # Have ResNet model take in grayscale rather than RGB
   model.conv1 = torch.nn.Conv2d(1, 64, kernel_size=7, stride=2, padding=3, bias=False)
   images, labels = next(iter(trainloader))
grid = torchvision.utils.make_grid(images)
   writer.add_image('images', grid, 0)
   writer.add_graph(model, images)
   writer.close()
This can then be visualized with TensorBoard, which should be
installable and runnable with:
pip install tensorboard
   tensorboard --logdir=runs
Lots of information can be logged for one experiment. To avoid
cluttering the UI and have better result clustering, we can group
plots by naming them hierarchically. For example, "Loss/train" and
"Loss/test" will be grouped together, while "Accuracy/train" and
"Accuracy/test" will be grouped separately in the TensorBoard
interface.
from torch.utils.tensorboard import SummaryWriter
   import numpy as np
import numpy as np
writer = SummaryWriter()
for n_iter in range(100):
       writer.add_scalar('Loss/train', np.random.random(), n_iter)
       writer.add_scalar('Loss/test', np.random.random(), n_iter)
       writer.add_scalar('Accuracy/train', np.random.random(), n_iter)
       writer.add_scalar('Accuracy/test', np.random.random(), n_iter)
Expected result:
[image]
class torch.utils.tensorboard.writer.SummaryWriter(log_dir=None, comment='', purge_step=None, max_queue=10, flush_secs=120, filename_suffix='')
Writes entries directly to event files in the log_dir to be
   consumed by TensorBoard.
The SummaryWriter class provides a high-level API to create an
   event file in a given directory and add summaries and events to it.
   The class updates the file contents asynchronously. This allows a
   training program to call methods to add data to the file directly
   from the training loop, without slowing down training.
init(log_dir=None, comment='', purge_step=None, max_queue=10, flush_secs=120, filename_suffix='')
  Creates a *SummaryWriter* that will write out events and
  summaries to the event file.

  Parameters:
     * **log_dir** (*str*) -- Save directory location. Default is
       runs/**CURRENT_DATETIME_HOSTNAME**, which changes after
       each run. Use hierarchical folder structure to compare
       between runs easily. e.g. pass in 'runs/exp1', 'runs/exp2',
       etc. for each new experiment to compare across them.

     * **comment** (*str*) -- Comment log_dir suffix appended to
       the default "log_dir". If "log_dir" is assigned, this
       argument has no effect.

     * **purge_step** (*int*) -- When logging crashes at step T+X
       and restarts at step T, any events whose global_step larger
       or equal to T will be purged and hidden from TensorBoard.
       Note that crashed and resumed experiments should have the
same "log_dir".
     * **max_queue** (*int*) -- Size of the queue for pending
       events and summaries before one of the 'add' calls forces a
       flush to disk. Default is ten items.

     * **flush_secs** (*int*) -- How often, in seconds, to flush
       the pending events and summaries to disk. Default is every
       two minutes.

     * **filename_suffix** (*str*) -- Suffix added to all event
       filenames in the log_dir directory. More details on
       filename construction in tensorboard.summary.writer.event_
       file_writer.EventFileWriter.

  Examples:

     from torch.utils.tensorboard import SummaryWriter

     # create a summary writer with automatically generated folder name.
     writer = SummaryWriter()
     # folder location: runs/May04_22-14-54_s-MacBook-Pro.local/

     # create a summary writer using the specified folder name.
     writer = SummaryWriter("my_experiment")
writer = SummaryWriter("my_experiment")
         # folder location: my_experiment
     # create a summary writer with comment appended.
     writer = SummaryWriter(comment="LR_0.1_BATCH_16")
     # folder location: runs/May04_22-14-54_s-MacBook-Pro.localLR_0.1_BATCH_16/

add_scalar(tag, scalar_value, global_step=None, walltime=None, new_style=False, double_precision=False)
  Add scalar data to summary.

  Parameters:
     * **tag** (*str*) -- Data identifier

     * **scalar_value** (*float** or **string/blobname*) -- Value
       to save

     * **global_step** (*int*) -- Global step value to record

     * **walltime** (*float*) -- Optional override default
       walltime (time.time()) with seconds after epoch of event

     * **new_style** (*boolean*) -- Whether to use new style
       (tensor field) or old style (simple_value field). New style
       could lead to faster data loading.

  Examples:
Examples:
     from torch.utils.tensorboard import SummaryWriter
     writer = SummaryWriter()
     x = range(100)
     for i in x:
         writer.add_scalar('y=2x', i * 2, i)
     writer.close()

  Expected result:

  [image]

add_scalars(main_tag, tag_scalar_dict, global_step=None, walltime=None)
  Adds many scalar data to summary.

  Parameters:
     * **main_tag** (*str*) -- The parent name for the tags

     * **tag_scalar_dict** (*dict*) -- Key-value pair storing the
       tag and corresponding values

     * **global_step** (*int*) -- Global step value to record

     * **walltime** (*float*) -- Optional override default
       walltime (time.time()) seconds after epoch of event

  Examples:

     from torch.utils.tensorboard import SummaryWriter
     writer = SummaryWriter()
     r = 5
     for i in range(100):
         writer.add_scalars('run_14h', {'xsinx':i*np.sin(i/r),
'xcosx':i*np.cos(i/r),
                                             'tanx': np.tan(i/r)}, i)
         writer.close()
         # This call adds three values to the same scalar plot with the tag
         # 'run_14h' in TensorBoard's scalar section.
  Expected result:

  [image]

add_histogram(tag, values, global_step=None, bins='tensorflow', walltime=None, max_bins=None)
  Add histogram to summary.

  Parameters:
     * **tag** (*str*) -- Data identifier

     * **values** (*torch.Tensor**, **numpy.ndarray**, or
       **string/blobname*) -- Values to build histogram

     * **global_step** (*int*) -- Global step value to record

     * **bins** (*str*) -- One of {'tensorflow','auto', 'fd',
       ...}. This determines how the bins are made. You can find
       other options in: https://docs.scipy.org/doc/numpy/referen
       ce/generated/numpy.histogram.html
ce/generated/numpy.histogram.html
     * **walltime** (*float*) -- Optional override default
       walltime (time.time()) seconds after epoch of event

  Examples:

     from torch.utils.tensorboard import SummaryWriter
     import numpy as np
     writer = SummaryWriter()
     for i in range(10):
         x = np.random.random(1000)
         writer.add_histogram('distribution centers', x + i, i)
     writer.close()

  Expected result:

  [image]

add_image(tag, img_tensor, global_step=None, walltime=None, dataformats='CHW')
  Add image data to summary.

  Note that this requires the "pillow" package.

  Parameters:
     * **tag** (*str*) -- Data identifier

     * **img_tensor** (*torch.Tensor**, **numpy.ndarray**, or
       **string/blobname*) -- Image data

     * **global_step** (*int*) -- Global step value to record

     * **walltime** (*float*) -- Optional override default
walltime (time.time()) seconds after epoch of event
     * **dataformats** (*str*) -- Image data format specification
       of the form CHW, HWC, HW, WH, etc.

  Shape:
     img_tensor: Default is (3, H, W). You can use
     "torchvision.utils.make_grid()" to convert a batch of tensor
     into 3xHxW format or call "add_images" and let us do the job.
     Tensor with (1, H, W), (H, W), (H, W, 3) is also suitable as
     long as corresponding "dataformats" argument is passed, e.g.
     "CHW", "HWC", "HW".

  Examples:

     from torch.utils.tensorboard import SummaryWriter
     import numpy as np
     img = np.zeros((3, 100, 100))
     img[0] = np.arange(0, 10000).reshape(100, 100) / 10000
     img[1] = 1 - np.arange(0, 10000).reshape(100, 100) / 10000

     img_HWC = np.zeros((100, 100, 3))
     img_HWC[:, :, 0] = np.arange(0, 10000).reshape(100, 100) / 10000
img_HWC[:, :, 1] = 1 - np.arange(0, 10000).reshape(100, 100) / 10000
     writer = SummaryWriter()
     writer.add_image('my_image', img, 0)

     # If you have non-default dimension setting, set the dataformats argument.
     writer.add_image('my_image_HWC', img_HWC, 0, dataformats='HWC')
     writer.close()

  Expected result:

  [image]

add_images(tag, img_tensor, global_step=None, walltime=None, dataformats='NCHW')
  Add batched image data to summary.

  Note that this requires the "pillow" package.

  Parameters:
     * **tag** (*str*) -- Data identifier

     * **img_tensor** (*torch.Tensor**, **numpy.ndarray**, or
       **string/blobname*) -- Image data

     * **global_step** (*int*) -- Global step value to record

     * **walltime** (*float*) -- Optional override default
       walltime (time.time()) seconds after epoch of event

     * **dataformats** (*str*) -- Image data format specification
of the form NCHW, NHWC, CHW, HWC, HW, WH, etc.
  Shape:
     img_tensor: Default is (N, 3, H, W). If "dataformats" is
     specified, other shape will be accepted. e.g. NCHW or NHWC.

  Examples:

     from torch.utils.tensorboard import SummaryWriter
     import numpy as np

     img_batch = np.zeros((16, 3, 100, 100))
     for i in range(16):
         img_batch[i, 0] = np.arange(0, 10000).reshape(100, 100) / 10000 / 16 * i
         img_batch[i, 1] = (1 - np.arange(0, 10000).reshape(100, 100) / 10000) / 16 * i

     writer = SummaryWriter()
     writer.add_images('my_image_batch', img_batch, 0)
     writer.close()

  Expected result:

  [image]

add_figure(tag, figure, global_step=None, close=True, walltime=None)
  Render matplotlib figure into an image and add it to summary.

  Note that this requires the "matplotlib" package.

  Parameters:
     * **tag** (*str*) -- Data identifier
tag (str) -- Data identifier * **figure** (*matplotlib.pyplot.figure*) -- Figure or a list
   of figures

 * **global_step** (*int*) -- Global step value to record

 * **close** (*bool*) -- Flag to automatically close the
   figure

 * **walltime** (*float*) -- Optional override default
   walltime (time.time()) seconds after epoch of event



add_video(tag, vid_tensor, global_step=None, fps=4, walltime=None)
  Add video data to summary.

  Note that this requires the "moviepy" package.

  Parameters:
     * **tag** (*str*) -- Data identifier

     * **vid_tensor** (*torch.Tensor*) -- Video data

     * **global_step** (*int*) -- Global step value to record

     * **fps** (*float** or **int*) -- Frames per second

     * **walltime** (*float*) -- Optional override default
       walltime (time.time()) seconds after epoch of event

  Shape:
Shape:
         vid_tensor: (N, T, C, H, W). The values should lie in [0,
         255] for type uint8 or [0, 1] for type float.
add_audio(tag, snd_tensor, global_step=None, sample_rate=44100, walltime=None)
  Add audio data to summary.

  Parameters:
     * **tag** (*str*) -- Data identifier

     * **snd_tensor** (*torch.Tensor*) -- Sound data

     * **global_step** (*int*) -- Global step value to record

     * **sample_rate** (*int*) -- sample rate in Hz

     * **walltime** (*float*) -- Optional override default
       walltime (time.time()) seconds after epoch of event

  Shape:
     snd_tensor: (1, L). The values should lie between [-1, 1].

add_text(tag, text_string, global_step=None, walltime=None)
  Add text data to summary.

  Parameters:
     * **tag** (*str*) -- Data identifier

     * **text_string** (*str*) -- String to save

     * **global_step** (*int*) -- Global step value to record
walltime (float) -- Optional override default
           walltime (time.time()) seconds after epoch of event
Examples:
 writer.add_text('lstm', 'This is an lstm', 0)
 writer.add_text('rnn', 'This is an rnn', 10)



add_graph(model, input_to_model=None, verbose=False, use_strict_trace=True)
  Add graph data to summary.

  Parameters:
     * **model** (*torch.nn.Module*) -- Model to draw.

     * **input_to_model** (*torch.Tensor** or **list of
       torch.Tensor*) -- A variable or a tuple of variables to be
       fed.

     * **verbose** (*bool*) -- Whether to print graph structure in
       console.

     * **use_strict_trace** (*bool*) -- Whether to pass keyword
       argument *strict* to *torch.jit.trace*. Pass False when you
       want the tracer to record your mutable container types
       (list, dict)

add_embedding(mat, metadata=None, label_img=None, global_step=None, tag='default', metadata_header=None)
Add embedding projector data to summary.
  Parameters:
     * **mat** (*torch.Tensor** or **numpy.ndarray*) -- A matrix
       which each row is the feature vector of the data point

     * **metadata** (*list*) -- A list of labels, each element
       will be convert to string

     * **label_img** (*torch.Tensor*) -- Images correspond to each
       data point

     * **global_step** (*int*) -- Global step value to record

     * **tag** (*str*) -- Name for the embedding

  Shape:
     mat: (N, D), where N is number of data and D is feature
     dimension

     label_img: (N, C, H, W)

  Examples:

     import keyword
     import torch
     meta = []
     while len(meta)<100:
         meta = meta+keyword.kwlist # get some strings
     meta = meta[:100]

     for i, v in enumerate(meta):
         meta[i] = v+str(i)

     label_img = torch.rand(100, 3, 10, 32)
     for i in range(100):