shrinath-suresh/blogs-docs-splitted huggingface dataset:
include  at::Tensor
  SigmoidAlphaBlendForwardCuda(....)

Instead of:
      #include  torch::Tensor
      SigmoidAlphaBlendForwardCuda(...)
Currently open issue for nvcc bug:
   https://github.com/pytorch/pytorch/issues/69460 Complete workaround
   code example: https://github.com/facebookresearch/pytorch3d/commit
   /cb170ac024a949f1f9614ffe6af1c38d972f7d48
Relocatable device code linking:
If you want to reference device symbols across compilation units
   (across object files), the object files need to be built with
   relocatable device code (-rdc=true or -dc). An exception to this
   rule is "dynamic parallelism" (nested kernel launches)  which is
   not used a lot anymore. Relocatable device code is less optimized
   so it needs to be used only on object files that need it. Using
   -dlto (Device Link Time Optimization) at the device code
   compilation step and dlink step help reduce the protentional perf
degradation of -rdc. Note that it needs to be used at both steps
   to be useful.
If you have rdc objects you need to have an extra -dlink
   (device linking) step before the CPU symbol linking step. There is
   also a case where -dlink is used without -rdc: when an
   extension is linked against a static lib containing rdc-compiled
   objects like the NVSHMEM
   library.
Note: Ninja is required to build a CUDA Extension with RDC linking.
-[ Example ]-



CUDAExtension(
   ...        name='cuda_extension',
   ...        sources=['extension.cpp', 'extension_kernel.cu'],
   ...        dlink=True,
   ...        dlink_libraries=["dlink_lib"],
   ...        extra_compile_args={'cxx': ['-g'],
   ...                            'nvcc': ['-O2', '-rdc=true']})



torch.utils.cpp_extension.BuildExtension(args, *kwargs)
A custom "setuptools" build extension .
This "setuptools.build_ext" subclass takes care of passing the
minimum required compiler flags (e.g. "-std=c++17") as well as
   mixed C++/CUDA compilation (and support for CUDA files in general).
When using "BuildExtension", it is allowed to supply a dictionary
   for "extra_compile_args" (rather than the usual list) that maps
   from languages ("cxx" or "nvcc") to a list of additional compiler
   flags to supply to the compiler. This makes it possible to supply
   different flags to the C++ and CUDA compiler during mixed
   compilation.
"use_ninja" (bool): If "use_ninja" is "True" (default), then we
   attempt to build using the Ninja backend. Ninja greatly speeds up
   compilation compared to the standard "setuptools.build_ext".
   Fallbacks to the standard distutils backend if Ninja is not
   available.
Note:
 By default, the Ninja backend uses #CPUS + 2 workers to build the
 extension. This may use up too many resources on some systems.
 One can control the number of workers by setting the *MAX_JOBS*
environment variable to a non-negative number.
torch.utils.cpp_extension.load(name, sources, extra_cflags=None, extra_cuda_cflags=None, extra_ldflags=None, extra_include_paths=None, build_directory=None, verbose=False, with_cuda=None, is_python_module=True, is_standalone=False, keep_intermediates=True)
Loads a PyTorch C++ extension just-in-time (JIT).
To load an extension, a Ninja build file is emitted, which is used
   to compile the given sources into a dynamic library. This library
   is subsequently loaded into the current Python process as a module
   and returned from this function, ready for use.
By default, the directory to which the build file is emitted and
   the resulting library compiled to is
   "/torch_extensions/", where "" is the temporary
   folder on the current platform and "" the name of the
   extension. This location can be overridden in two ways. First, if
   the "TORCH_EXTENSIONS_DIR" environment variable is set, it replaces
"/torch_extensions" and all extensions will be compiled into
   subfolders of this directory. Second, if the "build_directory"
   argument to this function is supplied, it overrides the entire
   path, i.e. the library will be compiled into that folder directly.
To compile the sources, the default system compiler ("c++") is
   used, which can be overridden by setting the "CXX" environment
   variable. To pass additional arguments to the compilation process,
   "extra_cflags" or "extra_ldflags" can be provided. For example, to
   compile your extension with optimizations, pass
   "extra_cflags=['-O3']". You can also use "extra_cflags" to pass
   further include directories.
CUDA support with mixed compilation is provided. Simply pass CUDA
   source files (".cu" or ".cuh") along with other sources. Such files
   will be detected and compiled with nvcc rather than the C++
   compiler. This includes passing the CUDA lib64 directory as a
library directory, and linking "cudart". You can pass additional
   flags to nvcc via "extra_cuda_cflags", just like with
   "extra_cflags" for C++. Various heuristics for finding the CUDA
   install directory are used, which usually work fine. If not,
   setting the "CUDA_HOME" environment variable is the safest option.
Parameters:
      * name -- The name of the extension to build. This MUST be
        the same as the name of the pybind11 module!
  * **sources** (*Union**[**str**, **List**[**str**]**]*) -- A
    list of relative or absolute paths to C++ source files.

  * **extra_cflags** -- optional list of compiler flags to forward
    to the build.

  * **extra_cuda_cflags** -- optional list of compiler flags to
    forward to nvcc when building CUDA sources.

  * **extra_ldflags** -- optional list of linker flags to forward
    to the build.

  * **extra_include_paths** -- optional list of include
    directories to forward to the build.
directories to forward to the build.
  * **build_directory** -- optional path to use as build
    workspace.

  * **verbose** -- If "True", turns on verbose logging of load
    steps.

  * **with_cuda** (*Optional**[**bool**]*) -- Determines whether
    CUDA headers and libraries are added to the build. If set to
    "None" (default), this value is automatically determined based
    on the existence of ".cu" or ".cuh" in "sources". Set it to
    *True`* to force CUDA headers and libraries to be included.

  * **is_python_module** -- If "True" (default), imports the
    produced shared library as a Python module. If "False",
    behavior depends on "is_standalone".

  * **is_standalone** -- If "False" (default) loads the
    constructed extension into the process as a plain dynamic
    library. If "True", build a standalone executable.

Returns:
      Returns the loaded PyTorch extension as a Python module.
If "is_python_module" is "False" and "is_standalone" is "False":
         Returns nothing. (The shared library is loaded into the
         process as a side effect.)
  If "is_standalone" is "True".
     Return the path to the executable. (On Windows,
     TORCH_LIB_PATH is added to the PATH environment variable as a
     side effect.)

Return type:
      If "is_python_module" is "True"
-[ Example ]-



from torch.utils.cpp_extension import load
module = load(
   ...     name='extension',
   ...     sources=['extension.cpp', 'extension_kernel.cu'],
   ...     extra_cflags=['-O2'],
   ...     verbose=True)



torch.utils.cpp_extension.load_inline(name, cpp_sources, cuda_sources=None, functions=None, extra_cflags=None, extra_cuda_cflags=None, extra_ldflags=None, extra_include_paths=None, build_directory=None, verbose=False, with_cuda=None, is_python_module=True, with_pytorch_error_handling=True, keep_intermediates=True)
Loads a PyTorch C++ extension just-in-time (JIT) from string
   sources.
This function behaves exactly like "load()", but takes its sources
   as strings rather than filenames. These strings are stored to files
   in the build directory, after which the behavior of "load_inline()"
   is identical to "load()".
See the tests for good examples of using this function.
Sources may omit two required parts of a typical non-inline C++
   extension: the necessary header includes, as well as the (pybind11)
   binding code. More precisely, strings passed to "cpp_sources" are
   first concatenated into a single ".cpp" file. This file is then
   prepended with "#include ".
Furthermore, if the "functions" argument is supplied, bindings will
   be automatically generated for each function specified. "functions"
   can either be a list of function names, or a dictionary mapping
   from function names to docstrings. If a list is given, the name of
each function is used as its docstring.
The sources in "cuda_sources" are concatenated into a separate
   ".cu" file and  prepended with "torch/types.h", "cuda.h" and
   "cuda_runtime.h" includes. The ".cpp" and ".cu" files are compiled
   separately, but ultimately linked into a single library. Note that
   no bindings are generated for functions in "cuda_sources" per  se.
   To bind to a CUDA kernel, you must create a C++ function that calls
   it, and either declare or define this C++ function in one of the
   "cpp_sources" (and include its name in "functions").
See "load()" for a description of arguments omitted below.
Parameters:
      * cpp_sources -- A string, or list of strings, containing
        C++ source code.
  * **cuda_sources** -- A string, or list of strings, containing
    CUDA source code.

  * **functions** -- A list of function names for which to
    generate function bindings. If a dictionary is given, it
should map function names to docstrings (which are otherwise
        just the function names).
  * **with_cuda** -- Determines whether CUDA headers and libraries
    are added to the build. If set to "None" (default), this value
    is automatically determined based on whether "cuda_sources" is
    provided. Set it to "True" to force CUDA headers and libraries
    to be included.

  * **with_pytorch_error_handling** -- Determines whether pytorch
    error and warning macros are handled by pytorch instead of
    pybind. To do this, each function "foo" is called via an
    intermediary "_safe_foo" function. This redirection might
    cause issues in obscure cases of cpp. This flag should be set
    to "False" when this redirect causes issues.

-[ Example ]-



from torch.utils.cpp_extension import load_inline
source = """
   at::Tensor sin_add(at::Tensor x, at::Tensor y) {
     return x.sin() + y.sin();
   }
   """
return x.sin() + y.sin();
   }
   """



module = load_inline(name='inline_extension',
   ...                      cpp_sources=[source],
   ...                      functions=['sin_add'])



Note:
 By default, the Ninja backend uses #CPUS + 2 workers to build the
 extension. This may use up too many resources on some systems.
 One can control the number of workers by setting the *MAX_JOBS*
 environment variable to a non-negative number.

torch.utils.cpp_extension.include_paths(cuda=False)
Get the include paths required to build a C++ or CUDA extension.
Parameters:
      cuda (bool) -- If True, includes CUDA-specific include
      paths.
Returns:
      A list of include path strings.
Return type:
      List[str]
torch.utils.cpp_extension.get_compiler_abi_compatibility_and_version(compiler)
Determine if the given compiler is ABI-compatible with PyTorch
   alongside its version.
Parameters:
alongside its version.
Parameters:
      compiler (str) -- The compiler executable name to check
      (e.g. "g++"). Must be executable in a shell process.
Returns:
      A tuple that contains a boolean that defines if the compiler is
      (likely) ABI-incompatible with PyTorch, followed by a
      TorchVersion string that contains the compiler version
      separated by dots.
Return type:
      Tuple[bool, TorchVersion]
torch.utils.cpp_extension.verify_ninja_availability()
Raises "RuntimeError" if ninja build system is not available on the
   system, does nothing otherwise.
torch.utils.cpp_extension.is_ninja_available()
Returns "True" if the ninja build system is available on the
   system, "False" otherwise.
Installing TorchDynamo
This section describes how to install TorchDynamo. TorchDynamo is
included in the nightly binaries of PyTorch. For more information, see
Getting Started.
Requirements
You must have the following prerequisites to use TorchDynamo:


A Linux or macOS environment


Python 3.8 (recommended). Python 3.7 through 3.10 are supported and
  tested. Make sure to have a development version of Python installed
  locally as well.


GPU/CUDA Requirements
To use GPU back ends, and in particular Triton, make sure that the
CUDA that you have installed locally matches the PyTorch version you
are running.
The following command installs GPU PyTorch + TorchDynamo along with
GPU TorchDynamo dependencies (for CUDA 11.7):
pip3 install numpy --pre torch[dynamo] --force-reinstall --extra-index-url https://download.pytorch.org/whl/nightly/cu117
CPU requirements
There are no additional requirements for CPU TorchDynamo. CPU
TorchDynamo is included in the nightly versions of PyTorch. To
install, run the following command:
pip3 install --pre torch --extra-index-url https://download.pytorch.org/whl/nightly/cpu
Install from Local Source
Alternatively, you can build PyTorch from source, which has
TorchDynamo included.
To install GPU TorchDynamo dependencies, run "make triton" in the
PyTorch repo root directory.
Verify Installation
If you built PyTorch from source, then you can run the following
commands (from the PyTorch repo root directory) to check that
TorchDynamo is installed correctly:
cd tools/dynamo
   python verify_dynamo.py
If you do not have the PyTorch source locally, you can alternatively
copy the script ("tools/dynamo/verify_dynamo.py") from the PyTorch
repository and run it locally.
Docker Installation
We also provide all the required dependencies in the PyTorch nightly
binaries which you can download with the following command:
TorchDynamo Overview
TorchDynamo is a Python-level JIT compiler designed to make
unmodified PyTorch programs faster. TorchDynamo hooks into the frame
evaluation API in CPython (PEP 523) to dynamically modify Python
bytecode right before it is executed. It rewrites Python bytecode in
order to extract sequences of PyTorch operations into an FX Graph
which is then just-in-time compiled with a customizable backend. It
creates this FX Graph through bytecode analysis and is designed to mix
Python execution with compiled backends to get the best of both worlds
â usability and performance.
TorchDynamo makes it easy to experiment with different compiler
backends to make PyTorch code faster with a single line decorator
"torch._dynamo.optimize()"
[image]
TorchInductor is one of the backends supported by TorchDynamo Graph
into Triton for GPUs or C++/OpenMP for CPUs. We have a training
performance dashboard that provides performance comparison for
Guards Overview
From a UX perspective, TorchDynamo is very easy to use. The user
invokes "torchdynamo.optimize" as an annotation:
@torchdynamo.optimize(my_compiler)
   def fn_foo(bar):
Where a complete example looks like this:
from typing import List
   import torch
   import torchdynamo
   def my_compiler(gm: torch.fx.GraphModule, example_inputs: List[torch.Tensor]):
       print("my_compiler() called with FX graph:")
       gm.graph.print_tabular()
       return gm.forward  # return a python callable
   @torchdynamo.optimize(my_compiler)
   def toy_example(a, b):
       x = a / (torch.abs(a) + 1)
       if b.sum() < 0:
           b = b * -1
       return x * b
   for _ in range(100):
       toy_example(torch.randn(10), torch.randn(10))
This allows TorchDynamo to capture the interpreted Python frames, grab
any and all relevant information, and speed things up wherever it can.
The speedup comes from a few places, and can be rather dependent on
the backend (my_compiler in the example above) provided, but the one
speedup that is important in this section is caching. Caching
itself is not a direct speedup but a critical enablement that prevents
recompilation. We dig a hole with dynamo, and caching allows us to get
out. It enables us to hold perf neutrality while then enabling
backends - the true source of our speedups.
With even a pass-through no-op backend provided:
def my_compiler(gm: torch.fx.GraphModule, example_inputs: List[torch.Tensor]):
       return gm.forward
We can see TorchDynamo speeding up Python execution even on regular
Python, not just PyTorch.
Caching and Guards Overview
TorchDynamo operates through caching transformed (by TorchDynamo) user
bytecode. When TorchDynamo receives a frame for evaluation, it checks
if the objects referenced in the frame have changed in certain
ways, and if not, TorchDynamo reads the previously transformed user
bytecode to evaluate it. In this section, we will focus on how we can
identify whether or not the objects referenced in the frame have
changed. This is a critical piece of functionality in TorchDynamo,
because it drives the entire invalidation lifecycle. This
functionality is called guards.
At a very high level, the flow can be summarized like this:


TorchDynamo receives a Python frame.


It converts the frame (1) passing it through instruction
   translation.


For the objects captured in (2), TorchDynamo creates tracking
   objects that are: * tracked on an output graph, which is an
   internal specialization of a torch.fx.Tracer * guards


TorchDynamo processes the guard objects created in (3), turning
   them into a generated Python function, check_fn, associated with
   a piece of code.


The check_fn is evaluated whenever we encounter this code a
   subsequent time - if a check_fn passes and evaluates to True,
   TorchDynamo identifies the code in the cache and the code
encountered here as same, and can be safely used. If it fails and
   evaluates to False, TorchDynamo identifies the code in the cache
   as not valid, and can be thrown out in favor of a new entry,
   through recompilation or a graph break.
Python Frame Evaluation and PEP 523
The functionality of TorchDynamo is based on PEP 523.
TorchDynamo installs a frame evaluation function on Python by using
_PyInterpreterState_SetEvalFrameFunc. TorchDynamo has a hook where
Python can hand control back to us during evaluation.
The function we have installed is "convert_frame" or
"convert_frame_assert" in the "nopython=True" case, but glossing over
that nuance for now, letâs take a look at "convert_frame_assert", as
"convert_frame" proxies to it.
We can find it on line 20 of convert_frame.py, with a signature as
follows:
def  convert_frame_assert(compiler_fn: Callable, one_graph=True):
This function wraps the entry point of where Python invokes
TorchDynamo with a frame:
def  _convert_frame_assert(frame: types.FrameType, cache_size: int):
Here is what this function does:


Checks if it has seen this "code"(see: f_code here) before and
   exits early if it did.


Checks if the code is an unsupported case.


Checks if the "cache_size" (second arg above) crosses the limit
   defined in the config, "cache_size_limit". If it has, the function
   drops the frame and logs warnings. This helps to avoid constant
   recompilation of a frame as it generally means that the frame is
   hot in an unexpected way and caching it produces needless overhead,
   as it is likely to get evicted the next time it is encountered.


Passes the frame, alongside a function that creates an
   "InstructionTranslator" through bytecode transformation, via
   "transform_code_object". A few crucial things happen under the hood
   here:


New code is produced through "transform_code_object".


An FX tracer named "output" is produced through
"InstructionTranslator".
  This can be a bit confusing, as "InstructionTranslator" is not
  an *fx* tracer, but its stored in a variable named tracer, and
  its output***is***an `fx`tracer.*



The function produces guards and stores them on "output" above.


The function produces "output_instructions" and stores them on
      "output" above.


The function maps the newly produced transformed code to the
      initial code it read off the frame. This mapping is worth
      remembering, we will refer to it much later on below where we
      cover guard failures.


Using the transformed code from 4.1 and the guards from 4.3, the
   function produces a GuardedCode.


Now that we have learned about frame evaluation, letâs review
"InstructionTranslator", and see how it turns the frame we handed it
over into TorchDynamo internal types.
InstructionTranslator
InstructionTranslator does a lot! We wonât cover the details of
everything it does, but most importantly for this document, it
produces a mapping of "symbolic_locals" which maintains a mapping from
the frameâs "f_locals" to TorchDynamo internal Variable objects (more
on these in a moment. "symbolic_locals" is filled via traversing the
frameâs locals:
self.symbolic_locals = collections.OrderedDict(
       (k, VariableBuilder(self, LocalSource(k))(f_locals[k]))
       for k in vars
       if k in f_locals
   )
The important component here  is the invocation of a call into
"VariableBuilder". "VariableBuilder"âs call implementation proxies
into a function called "_wrap", which in turn both constructs
instances of "VariableTracker" and calls "make_guards" on them. More
on that later.
This mapping, in turn, is critical as each Variable has associated
guards, which are then passed to "self.output", the instance of
"OutputGraph", an fx tracer, mentioned in 4.2 of the section above. If
you recall, this "OutputGraph", stored in a variable called "output"
is where our guards are stored before being passed on to become
"GuardedCode"
How does "InstructionTranslator" do this? At the heart of it, there is
a loop that is pumped, which drives a function "step".
"step" is just that - a single processing step, taking exactly one
instruction and doing something with it.
Note:
These are real instructions processed by TorchDynamoâs
  "transform_code_object", and it is pretty cool.
Note:
This section purposely skips the details of dis.get_instructions.
For the example above, here is a snippet of a what a few
"Instruction"'s may look like:
Instruction(opcode=124, opname='LOAD_FAST', arg=0, argval='b', offset=32, starts_line=8, is_jump_target=True, target=None)
   Instruction(opcode=100, opname='LOAD_CONST', arg=3, argval=-1, offset=34, starts_line=None, is_jump_target=False, target=None)
   Instruction(opcode=20, opname='BINARY_MULTIPLY', arg=None, argval=None, offset=36, starts_line=None, is_jump_target=False, target=None)
This is the core functionality of this function. Take a look at the
"opname", and then take a look at this little snippet from inside
"step";
if not hasattr(self, inst.opname):
       unimplemented(f"missing: {inst.opname}")
   getattr(self, inst.opname)(inst)
As we can see, the function checks if the current class, the
"InstructionTranslator" has an attribute set matching the operator
name (for example, "LOAD_CONST"). If it does, the function invokes it,
passing the whole instruction object in. If it does not, the function
drops the frame as unimplemented.
For the "LOAD_CONST" example, we can see that we do indeed support it,
with a relatively straightforward definition:
def  LOAD_CONST(self, inst):
   self.push(ConstantVariable(value=inst.argval))
We can see that this function creates a new instance of the class
"ConstantVariable" , with a value, in our example case, -1, and then
pushes it onto the stack.
There are dozens of such methods - see "symbolic_convert.py" for all
of them. Generally, we implement as many matching methods to Python
bytecode instructions as possible.
Across both the logic downstream of "step" and the logic from invoking
"VariableBuilder" - we now have a lot of "VariableTracker"s and of
course, weâve spoken about creating guards quiet a bit. Letâs dig into
what Variables are, and get a little closer to understanding guards.
Variables
A "ConstantVariable" is an instance of"VariableTracker".
"VariableTracker" represents a tracked Python local or stack value.
When it comes to representing an object inside TorchDynamo, a
"VariableTracker" does exactly what it says - it tracks a given
variable. It is an extremely flexible class, but there are a few
points to keep in mind:


It manages the "guard" relationship around the underlying object
  through:


"make_guard"


"replace_guards"


"add_guard(s)"


"propagate" - "propagate(*vars: List[List["VariableTracker"]])" -