shrinath-suresh/blogs-docs-splitted huggingface dataset:
Prepare the model by attaching the logger to both float module and
   quantized module if they are in the allow_list.
Parameters:
      * float_module (Module) -- float module used to generate
        the q_module
  * **q_module** (*Module*) -- module quantized from float_module

  * **logger_cls** -- type of logger to be attached to
    float_module and q_module

  * **allow_list** -- list of module types to attach logger

torch.ao.ns._numeric_suite.compare_model_outputs(float_model, q_model, *data, logger_cls=, allow_list=None)
Compare output activations between float and quantized models at
   corresponding locations for the same input. Return a dict with key
   corresponding to quantized module names and each entry being a
   dictionary with two keys 'float' and 'quantized', containing the
   activations of quantized model and float model at matching
   locations. This dict can be used to compare and compute the
propagation quantization error.
Example usage:
  act_compare_dict = compare_model_outputs(float_model, qmodel, data)
  for key in act_compare_dict:
      print(
          key,
          compute_error(
              act_compare_dict[key]['float'],
              act_compare_dict[key]['quantized'].dequantize()
          )
      )

Parameters:
      * float_model (Module) -- float model used to generate the
        q_model
  * **q_model** (*Module*) -- model quantized from float_model

  * **data** -- input data used to run the prepared float_model
    and q_model

  * **logger_cls** -- type of logger to be attached to
    float_module and q_module

  * **allow_list** -- list of module types to attach logger

Returns:
      dict with key corresponding to quantized module names and each
      entry being a dictionary with two keys 'float' and 'quantized',
      containing the matching float and quantized activations
Return type:
torch.utils.model_zoo
Moved to torch.hub.
torch.utils.model_zoo.load_url(url, model_dir=None, map_location=None, progress=True, check_hash=False, file_name=None)
Loads the Torch serialized object at the given URL.
If downloaded file is a zip file, it will be automatically
   decompressed.
If the object is already present in model_dir, it's deserialized
   and returned. The default value of "model_dir" is
   "/checkpoints" where "hub_dir" is the directory returned
   by "get_dir()".
Parameters:
      * url (str) -- URL of the object to download
  * **model_dir** (*str**, **optional*) -- directory in which to
    save the object

  * **map_location** (*optional*) -- a function or a dict
    specifying how to remap storage locations (see torch.load)

  * **progress** (*bool**, **optional*) -- whether or not to
    display a progress bar to stderr. Default: True
check_hash (bool, optional) -- If True, the filename
        part of the URL should follow the naming convention
        "filename-.ext" where "" is the first eight or
        more digits of the SHA256 hash of the contents of the file.
        The hash is used to ensure unique names and to verify the
        contents of the file. Default: False

file_name (str, optional) -- name for the downloaded
    file. Filename from "url" will be used if not set.



Return type:
      Dict[str, Any]
-[ Example ]-



state_dict = torch.hub.load_state_dict_from_url('https://s3.amazonaws.com/pytorch/models/resnet18-5c106cde.pth')
Warning:
There are known non-determinism issues for RNN functions on some
  versions of cuDNN and CUDA. You can enforce deterministic behavior
  by setting the following environment variables:On CUDA 10.1, set
  environment variable "CUDA_LAUNCH_BLOCKING=1". This may affect
  performance.On CUDA 10.2 or later, set environment variable (note
  the leading colon symbol) "CUBLAS_WORKSPACE_CONFIG=:16:8" or
  "CUBLAS_WORKSPACE_CONFIG=:4096:2".See the cuDNN 8 Release Notes for
  more information.
PyTorch documentation
PyTorch is an optimized tensor library for deep learning using GPUs
and CPUs.
Features described in this documentation are classified by release
status:
Stable:  These features will be maintained long-term and there
   should generally be no major performance limitations or gaps in
   documentation. We also expect to maintain backwards compatibility
   (although breaking changes can happen and notice will be given one
   release ahead of time).
Beta:  These features are tagged as Beta because the API may
   change based on user feedback, because the performance needs to
   improve, or because coverage across operators is not yet complete.
   For Beta features, we are committing to seeing the feature through
   to the Stable classification. We are not, however, committing to
   backwards compatibility.
Prototype:  These features are typically not available as part of
   binary distributions like PyPI or Conda, except sometimes behind
run-time flags, and are at an early stage for feedback and testing.
Community
^^^^^^^^^


PyTorch Governance | Build + CI


PyTorch Contribution Guide


PyTorch Design Philosophy


PyTorch Governance | Mechanics


PyTorch Governance | Maintainers


Developer Notes
^^^^^^^^^^^^^^^


CUDA Automatic Mixed Precision examples


Autograd mechanics


Broadcasting semantics


CPU threading and TorchScript inference


CUDA semantics


Distributed Data Parallel


Extending PyTorch


Extending torch.func with autograd.Function


Frequently Asked Questions


Gradcheck mechanics


HIP (ROCm) semantics


Features for large-scale deployments


Modules


MPS backend


Multiprocessing best practices


Numerical accuracy


Reproducibility


Serialization semantics


Windows FAQ


Language Bindings
^^^^^^^^^^^^^^^^^


C++


Javadoc


torch::deploy


Python API
^^^^^^^^^^


torch


Tensors


Generators


Random sampling


Serialization


Parallelism
Serialization


Parallelism


Locally disabling gradient computation


Math operations


Utilities


Symbolic Numbers


Optimizations


Operator Tags


Engine Configuration


torch.nn


Parameter


UninitializedParameter


UninitializedBuffer


Containers


Convolution Layers


Pooling layers


Padding Layers


Non-linear Activations (weighted sum, nonlinearity)


Non-linear Activations (other)


Normalization Layers


Recurrent Layers


Transformer Layers


Linear Layers


Dropout Layers


Sparse Layers


Distance Functions


Loss Functions


Vision Layers


Shuffle Layers


DataParallel Layers (multi-GPU, distributed)


Utilities


Quantized Functions


Lazy Modules Initialization


torch.nn.functional


Convolution functions


Pooling functions


Non-linear activation functions


Linear functions


Dropout functions


Sparse functions


Distance functions


Loss functions
Distance functions


Loss functions


Vision functions


DataParallel functions (multi-GPU, distributed)


torch.Tensor


Data types


Initializing and basic operations


Tensor class reference


Tensor Attributes


torch.dtype


torch.device


torch.layout


torch.memory_format


Tensor Views


torch.amp


Autocasting


Gradient Scaling


Autocast Op Reference


torch.autograd


torch.autograd.backward


torch.autograd.grad


Forward-mode Automatic Differentiation


Functional higher level API


Locally disabling gradient computation


Default gradient layouts


In-place operations on Tensors


Variable (deprecated)


Tensor autograd functions


Function


Context method mixins


Numerical gradient checking


Profiler


Anomaly detection


Autograd graph


torch.library


torch.cuda


StreamContext


torch.cuda.can_device_access_peer


torch.cuda.current_blas_handle


torch.cuda.current_device
torch.cuda.current_device


torch.cuda.current_stream


torch.cuda.default_stream


device


torch.cuda.device_count


device_of


torch.cuda.get_arch_list


torch.cuda.get_device_capability


torch.cuda.get_device_name


torch.cuda.get_device_properties


torch.cuda.get_gencode_flags


torch.cuda.get_sync_debug_mode


torch.cuda.init


torch.cuda.ipc_collect


torch.cuda.is_available


torch.cuda.is_initialized


torch.cuda.memory_usage


torch.cuda.set_device


torch.cuda.set_stream


torch.cuda.set_sync_debug_mode


torch.cuda.stream


torch.cuda.synchronize


torch.cuda.utilization


torch.cuda.OutOfMemoryError


Random Number Generator


Communication collectives


Streams and events


Graphs (beta)


Memory management


NVIDIA Tools Extension (NVTX)


Jiterator (beta)


Stream Sanitizer (prototype)


torch.backends


torch.backends.cuda


torch.backends.cudnn


torch.backends.mps
torch.backends.cudnn


torch.backends.mps


torch.backends.mkl


torch.backends.mkldnn


torch.backends.openmp


torch.backends.opt_einsum


torch.backends.xeon


torch.distributed


Backends


Basics


Initialization


Post-Initialization


Distributed Key-Value Store


Groups


Point-to-point communication


Synchronous and asynchronous collective operations


Collective functions


Profiling Collective Communication


Multi-GPU collective functions


Third-party backends


Launch utility


Spawn utility


Debugging "torch.distributed" applications


Logging


torch.distributed.algorithms.join


torch.distributed.elastic


Get Started


Documentation


torch.distributed.fsdp


torch.distributed.optim


torch.distributed.tensor.parallel


torch.distributed.checkpoint


torch.distributions


Score function


Pathwise derivative


Distribution


ExponentialFamily


Bernoulli


Beta


Binomial
Bernoulli


Beta


Binomial


Categorical


Cauchy


Chi2


ContinuousBernoulli


Dirichlet


Exponential


FisherSnedecor


Gamma


Geometric


Gumbel


HalfCauchy


HalfNormal


Independent


Kumaraswamy


LKJCholesky


Laplace


LogNormal


LowRankMultivariateNormal


MixtureSameFamily


Multinomial


MultivariateNormal


NegativeBinomial


Normal


OneHotCategorical


Pareto


Poisson


RelaxedBernoulli


LogitRelaxedBernoulli


RelaxedOneHotCategorical


StudentT


TransformedDistribution


Uniform


VonMises


Weibull


Wishart


KL Divergence


Transforms


Constraints


Constraint Registry


torch._dynamo


torch.fft


Fast Fourier Transforms


Helper Functions


torch.func


What are composable function transforms?


Why composable function transforms?


Read More


torch.futures


torch.fx


Overview


Writing Transformations


Debugging
Writing Transformations


Debugging


Limitations of Symbolic Tracing


API Reference


torch.hub


Publishing models


Loading models from Hub


torch.jit


TorchScript Language Reference


Creating TorchScript Code


Mixing Tracing and Scripting


TorchScript Language


Built-in Functions and Modules


Debugging


Frequently Asked Questions


Known Issues


Appendix


torch.linalg


Matrix Properties


Decompositions


Solvers


Inverses


Matrix Functions


Matrix Products


Tensor Operations


Misc


Experimental Functions


torch.monitor


API Reference


torch.signal


torch.signal.windows


torch.special


Functions


torch.overrides


Functions


torch.package


Tutorials


How do I...


Explanation


API Reference


torch.profiler


Overview


API Reference


Intel Instrumentation and Tracing Technology APIs


torch.nn.init


torch.onnx


Example: AlexNet from PyTorch to ONNX
Example: AlexNet from PyTorch to ONNX


Tracing vs Scripting


Avoiding Pitfalls


Limitations


Adding support for operators


Frequently Asked Questions


Contributing / developing


Functions


Classes


torch.onnx diagnostics


Overview


Diagnostic Rules


API Reference


torch.optim


How to use an optimizer


Base class


Algorithms


How to adjust learning rate


Stochastic Weight Averaging


Complex Numbers


Creating Complex Tensors


Transition from the old representation


Accessing real and imag


Angle and abs


Linear Algebra


Serialization


Autograd


DDP Communication Hooks


How to Use a Communication Hook?


What Does a Communication Hook Operate On?


Default Communication Hooks


PowerSGD Communication Hook


Debugging Communication Hooks


Checkpointing of Communication Hooks


Acknowledgements


Pipeline Parallelism


Model Parallelism using multiple GPUs


Pipelined Execution
Pipelined Execution


Pipe APIs in PyTorch


Tutorials


Acknowledgements


Quantization


Introduction to Quantization


Quantization API Summary


Quantization Stack


Quantization Support Matrix


Quantization API Reference


Quantization Backend Configuration


Quantization Accuracy Debugging


Quantization Customizations


Best Practices


Frequently Asked Questions


Common Errors


Distributed RPC Framework


Basics


RPC


RRef


RemoteModule


Distributed Autograd Framework


Distributed Optimizer


Design Notes


Tutorials


torch.random


torch.masked


Introduction


Supported Operators


torch.nested


Introduction


Construction


size


unbind


Nested tensor constructor and conversion functions


Supported operations


torch.sparse


Why and when to use sparsity


Functionality overview


Operator overview


Sparse COO tensors


Sparse Compressed Tensors


Supported operations
Supported operations


torch.Storage


torch.testing


torch.utils.benchmark


torch.utils.bottleneck


torch.utils.checkpoint


torch.utils.cpp_extension


torch.utils.data


Dataset Types


Data Loading Order and "Sampler"


Loading Batched and Non-Batched Data


Single- and Multi-process Data Loading


Memory Pinning


torch.utils.jit


torch.utils.dlpack


torch.utils.mobile_optimizer


torch.utils.model_zoo


torch.utils.tensorboard


Type Info


torch.finfo


torch.iinfo


Named Tensors


Creating named tensors


Named dimensions


Name propagation semantics


Explicit alignment by names


Manipulating dimensions


Autograd support


Currently supported operations and subsystems


Named tensor API reference


Named Tensors operator coverage


Keeps input names


Removes dimensions


Unifies names from inputs


Permutes dimensions


Contracts away dims


Factory functions


out function and in-place variants
Events
Module contains events processing mechanisms that are integrated with
the standard python logging.
Example of usage:
from torch.distributed.elastic import events
   event = events.Event(name="test_event", source=events.EventSource.WORKER, metadata={...})
   events.get_logging_handler(destination="console").info(event)
API Methods
torch.distributed.elastic.events.record(event, destination='null')
torch.distributed.elastic.events.get_logging_handler(destination='null')
Return type:
      Handler
Event Objects
class torch.distributed.elastic.events.api.Event(name, source, timestamp=0, metadata=)
The class represents the generic event that occurs during the
   torchelastic job execution. The event can be any kind of meaningful
   action.
Parameters:
      * name (str) -- event name.
  * **source** (*EventSource*) -- the event producer, e.g. agent
    or worker
or worker
  * **timestamp** (*int*) -- timestamp in milliseconds when event
    occured.

  * **metadata** (*Dict**[**str**, **Optional**[**Union**[**str**,
    **int**, **float**, **bool**]**]**]*) -- additional data that
    is associated with the event.

class torch.distributed.elastic.events.api.EventSource(value)
Known identifiers of the event producers.
torch.distributed.elastic.events.api.EventMetadataValue
alias of "Optional"["Union"["str", "int", "float", "bool"]]
Metrics
Metrics API
Overview:
The metrics API in torchelastic is used to publish telemetry metrics.
It is designed to be used by torchelastic's internal modules to
publish metrics for the end user with the goal of increasing
visibility and helping with debugging. However you may use the same
API in your jobs to publish metrics to the same metrics "sink".
A "metric" can be thought of as timeseries data and is uniquely
identified by the string-valued tuple "(metric_group, metric_name)".
torchelastic makes no assumptions about what a "metric_group" is and
what relationship it has with "metric_name". It is totally up to the
user to use these two fields to uniquely identify a metric.
Note:
The metric group "torchelastic" is reserved by torchelastic for
  platform level metrics that it produces. For instance torchelastic
  may output the latency (in milliseconds) of a re-rendezvous
  operation from the agent as "(torchelastic,
  agent.rendezvous.duration.ms)"
agent.rendezvous.duration.ms)"
A sensible way to use metric groups is to map them to a stage or
module in your job. You may also encode certain high level properties
the job such as the region or stage (dev vs prod).
Publish Metrics:
Using torchelastic's metrics API is similar to using python's logging
framework. You first have to configure a metrics handler before trying
to add metric data.
The example below measures the latency for the "calculate()" function.
import time
   import torch.distributed.elastic.metrics as metrics
# makes all metrics other than the one from "my_module" to go /dev/null
   metrics.configure(metrics.NullMetricsHandler())
   metrics.configure(metrics.ConsoleMetricsHandler(), "my_module")
def my_method():
     start = time.time()
     calculate()
     end = time.time()
     metrics.put_metric("calculate_latency", int(end-start), "my_module")
You may also use the torch.distributed.elastic.metrics.prof` decorator
to conveniently and succinctly profile functions
to conveniently and succinctly profile functions
# -- in module examples.foobar --
import torch.distributed.elastic.metrics as metrics
metrics.configure(metrics.ConsoleMetricsHandler(), "foobar")
   metrics.configure(metrics.ConsoleMetricsHandler(), "Bar")
@metrics.prof
   def foo():
     pass
class Bar():
 @metrics.prof
 def baz():
     pass

"@metrics.prof" will publish the following metrics
.success - 1 if the function finished successfully
   .failure - 1 if the function threw an exception
   .duration.ms - function duration in milliseconds
Configuring Metrics Handler:
torch.distributed.elastic.metrics.MetricHandler is responsible for
emitting the added metric values to a particular destination. Metric
groups can be configured with different metric handlers.
By default torchelastic emits all metrics to "/dev/null". By adding
the following configuration metrics, "torchelastic" and "my_app"
metric groups will be printed out to console.
import torch.distributed.elastic.metrics as metrics
metrics.configure(metrics.ConsoleMetricHandler(), group = "torchelastic")
   metrics.configure(metrics.ConsoleMetricHandler(), group = "my_app")
Writing a Custom Metric Handler:
If you want your metrics to be emitted to a custom location, implement
the torch.distributed.elastic.metrics.MetricHandler interface and
configure your job to use your custom metric handler.
Below is a toy example that prints the metrics to "stdout"
import torch.distributed.elastic.metrics as metrics
class StdoutMetricHandler(metrics.MetricHandler):
      def emit(self, metric_data):
          ts = metric_data.timestamp
          group = metric_data.group_name
          name = metric_data.name
          value = metric_data.value
          print(f"[{ts}][{group}]: {name}={value}")
metrics.configure(StdoutMetricHandler(), group="my_app")
Now all metrics in the group "my_app" will be printed to stdout as:
[1574213883.4182858][my_app]: my_metric=
   [1574213940.5237644][my_app]: my_metric=
Metric Handlers
Below are the metric handlers that come included with torchelastic.
class torch.distributed.elastic.metrics.api.MetricHandler
class torch.distributed.elastic.metrics.api.ConsoleMetricHandler
class torch.distributed.elastic.metrics.api.NullMetricHandler
Methods
torch.distributed.elastic.metrics.configure(handler, group=None)
torch.distributed.elastic.metrics.prof(fn=None, group='torchelastic')
@profile decorator publishes duration.ms, count, success, failure
   metrics for the function that it decorates. The metric name
   defaults to the qualified name ("class_name.def_name") of the
   function. If the function does not belong to a class, it uses the
   leaf module name instead.
Usage
  @metrics.prof
  def x():
      pass

  @metrics.prof(group="agent")
Quickstart
To launch a fault-tolerant job, run the following on all nodes.
torchrun
      --nnodes=NUM_NODES
      --nproc_per_node=TRAINERS_PER_NODE
      --max_restarts=NUM_ALLOWED_FAILURES
      --rdzv_id=JOB_ID
      --rdzv_backend=c10d
      --rdzv_endpoint=HOST_NODE_ADDR
      YOUR_TRAINING_SCRIPT.py (--arg1 ... train script args...)
To launch an elastic job, run the following on at least "MIN_SIZE"
nodes and at most "MAX_SIZE" nodes.
torchrun
       --nnodes=MIN_SIZE:MAX_SIZE
       --nproc_per_node=TRAINERS_PER_NODE
       --max_restarts=NUM_ALLOWED_FAILURES_OR_MEMBERSHIP_CHANGES
       --rdzv_id=JOB_ID
       --rdzv_backend=c10d
       --rdzv_endpoint=HOST_NODE_ADDR
       YOUR_TRAINING_SCRIPT.py (--arg1 ... train script args...)
Note:
TorchElastic models failures as membership changes. When a node
  fails, this is treated as a "scale down" event. When the failed node
  is replaced by the scheduler, it is a "scale up" event. Hence for
both fault tolerant and elastic jobs, "--max_restarts" is used to
  control the total number of restarts before giving up, regardless of
  whether the restart was caused due to a failure or a scaling event.
"HOST_NODE_ADDR", in form [:] (e.g.
node1.example.com:29400), specifies the node and the port on which the
C10d rendezvous backend should be instantiated and hosted. It can be
any node in your training cluster, but ideally you should pick a node
that has a high bandwidth.
Note:
If no port number is specified "HOST_NODE_ADDR" defaults to 29400.
Note:
The "--standalone" option can be passed to launch a single node job
  with a sidecar rendezvous backend. You donât have to pass "--
  rdzv_id", "--rdzv_endpoint", and "--rdzv_backend" when the "--
  standalone" option is used.
Note:
Learn more about writing your distributed training script here.
If "torchrun" does not meet your requirements you may use our APIs
directly for more powerful customization. Start by taking a look at
Customization
This section describes how to customize TorchElastic to fit your
needs.
Launcher
The launcher program that ships with TorchElastic should be sufficient
for most use-cases (see torchrun (Elastic Launch)). You can implement
a custom launcher by programmatically creating an agent and passing it
specs for your workers as shown below.
# my_launcher.py
if name == "main":
     args = parse_args(sys.argv[1:])
     rdzv_handler = RendezvousHandler(...)
     spec = WorkerSpec(
         local_world_size=args.nproc_per_node,
         fn=trainer_entrypoint_fn,
         args=(trainer_entrypoint_fn args.fn_args,...),
         rdzv_handler=rdzv_handler,
         max_restarts=args.max_restarts,
         monitor_interval=args.monitor_interval,
     )
 agent = LocalElasticAgent(spec, start_method="spawn")
 try:
     run_result = agent.run()
     if run_result.is_failed():
         print(f"worker 0 failed with: run_result.failures[0]")
     else:
else:
             print(f"worker 0 return value is: run_result.return_values[0]")
     except Exception ex:
         # handle exception
Rendezvous Handler
To implement your own rendezvous, extend
"torch.distributed.elastic.rendezvous.RendezvousHandler" and implement
its methods.
Warning:
Rendezvous handlers are tricky to implement. Before you begin make
  sure you completely understand the properties of rendezvous. Please
  refer to Rendezvous for more information.
Once implemented you can pass your custom rendezvous handler to the
worker spec when creating the agent.
spec = WorkerSpec(
       rdzv_handler=MyRendezvousHandler(params),
       ...
   )
   elastic_agent = LocalElasticAgent(spec, start_method=start_method)
   elastic_agent.run(spec.role)
Metric Handler
TorchElastic emits platform level metrics (see Metrics). By default
metrics are emitted to /dev/null so you will not see them. To have