shrinath-suresh/blogs-docs-splitted huggingface dataset:
PathAI develops machine learning models which provide insights for drug development R&D, for powering clinical trials, and for making diagnoses. To this end, PathAI leverages PyTorch for slide level inference using a variety of methods including graph neural networks (GNN) as well as multiple instance learning. In this context, “slides” refers to full size scanned images of glass slides, which are pieces of glass with a thin slice of tissue between them, stained to show various cell formations. PyTorch enables our teams using these different methodologies to share a common framework which is robust enough to work in all the conditions we need. PyTorch’s high level, imperative, and pythonic syntax allows us to prototype models quickly and then take those models to scale once we have the results we want. 
Multi-instance learning on gigabyte images
One of the uniquely challenging aspects of applying ML to pathology is the immense size of the images. These digital slides can often be 100,000 x 100,000 pixels or more in resolution and gigabytes in size. Loading the full image in GPU memory and applying traditional computer vision algorithms on them is an almost impossible task. It also takes both a considerable amount of time and resources to have a full slide image (100k x 100k) annotated, especially when annotators need to be domain experts (board-certified pathologists). We often build models to predict image-level labels, like the presence of cancer, on a patient slide which covers a few thousand pixels in the whole image. The cancerous area is sometimes a tiny fraction of the entire slide, which makes the ML problem similar to finding a needle in a haystack. On the other hand, some problems like the prediction of certain histological biomarkers require an aggregation of information from the whole slide which is again hard due to the size of the images. All these factors add significant algorithmic, computational, and logistical complexity when applying ML techniques to pathology problems.
Breaking down the image into smaller patches, learning patch representations, and then pooling those representations to predict an image-level label is one way to solve this problem as is depicted in the image below. One popular method for doing this is called Multiple Instance Learning (MIL). Each patch is considered an ‘instance’ and a set of patches forms a ‘bag’. The individual patch representations are pooled together to predict a final bag-level label. Algorithmically, the individual patch instances in the bag do not require labels and hence allow us to learn bag-level labels in a weakly-supervised way. They also use permutation invariant pooling functions which make the prediction independent of the order of patches and allows for an efficient aggregation of information. Typically, attention based pooling functions are used which not only allow for efficient aggregation but also provide attention values for each patch in the bag. These values indicate the importance of the corresponding patch in the prediction and can be visualized to better understand the model predictions. This element of interpretability can be very important to drive adoption of these models in the real world and we use variations like Additive MIL models to enable such spatial explainability. Computationally, MIL models circumvent the problem of applying neural networks to large image sizes since patch representations are obtained independently of the size of the image.
At PathAI, we use custom MIL models based on deep nets to predict image-level labels. The overview of this process is as follows:

Select patches from a slide using different sampling approaches.
Construct a bag of patches based on random sampling or heuristic rules.
Generate patch representations for each instance based on pre-trained models or large-scale representation learning models.
Apply permutation invariant pooling functions to get the final slide-level score.

Now that we have walked through some of the high-level details around MIL in PyTorch, let’s look at some code to see how simple it is to go from ideation to code in production with PyTorch. We begin by defining a sampler, transformations, and our MIL dataset:
```python
Create a bag sampler which randomly samples patches from a slide
bag_sampler = RandomBagSampler(bag_size=12)
Setup the transformations
crop_transform = FlipRotateCenterCrop(use_flips=True)
Create the dataset which loads patches for each bag
train_dataset = MILDataset(
  bag_sampler=bag_sampler,
  samples_loader=sample_loader,
  transform=crop_transform,
)

After we have defined our sampler and dataset, we need to define the model we will actually train with said dataset. PyTorch’s familiar model definition syntax makes this easy to do while also allowing us to create bespoke models at the same time.

```python
classifier = DefaultPooledClassifier(hidden_dims=[256, 256], input_dims=1024, output_dims=1)

pooling = DefaultAttentionModule(
  input_dims=1024,
  hidden_dims=[256, 256],
  output_activation=StableSoftmax()
)

# Define the model which is a composition of the featurizer, pooling module and a classifier
model = DefaultMILGraph(featurizer=ShuffleNetV2(), classifier=classifier, pooling = pooling)
```
Since these models are trained end-to-end, they offer a powerful way to go directly from a gigapixel whole slide image to a single label. Due to their wide applicability to different biological problems, two aspects of their implementation and deployment are important:

Configurable control over each part of the pipeline including the data loaders, the modular parts of the model, and their interaction with each other.
Ability to rapidly iterate through the ideate-implement-experiment-productionize loop.
PyTorch has various advantages when it comes to MIL modeling. It offers an intuitive way to create dynamic computational graphs with flexible control flow which is great for rapid research experimentation. The map-style datasets, configurable sampler and batch-samplers allow us to customize how we construct bags of patches, enabling faster experimentation. Since MIL models are IO heavy, data parallelism and pythonic data loaders make the task very efficient and user friendly. Lastly, the object-oriented nature of PyTorch enables building of reusable modules which aid in the rapid experimentation, maintainable implementation and ease of building compositional components of the pipeline.
Exploring spatial tissue organization with GNNs in PyTorch
In both healthy and diseased tissue, the spatial arrangement and structure of cells can oftentimes be as important as the cells themselves. For example, when assessing lung cancers, pathologists try to look at the overall grouping and structure of tumor cells (do they form solid sheets? Or do they occur in smaller, localized clusters?) to determine if the cancer belongs to specific subtypes which can have vastly different prognosis. Such spatial relationships between cells and other tissue structures can be modeled using graphs to capture tissue topology and cellular composition at the same time. Graph Neural Networks (GNNs) allow learning spatial patterns within these graphs that relate to other clinical variables, for example overexpression of genes in certain cancers.
One of the main value-adds of GNN’s in the context of tissue samples is that the graph itself can uncover spatial relationships that would otherwise be very difficult to find by visual inspection alone. In our recent AACR publication, we showed that by using GNNs, we can better understand the way the presence of immune cell aggregates (specifically tertiary lymphoid structures, or TLS) in the tumor microenvironment can influence patient prognosis. In this case, the GNN approach was used to predict expression of genes associated with the presence of TLS, and identify histological features beyond the TLS region itself that are relevant to TLS. Such insights into gene expression are difficult to identify from tissue sample images when unassisted by ML models.
One of the most promising GNN variations we have had success with is self attention graph pooling. Let’s take a look at how we define our Self Attention Graph Pooling (SAGPool) model using PyTorch and PyG:
class SAGPool(torch.nn.Module):
  def __init__(self, ...):
    super().__init__()
    self.conv1 = GraphConv(in_features, hidden_features, aggr='mean')
    self.convs = torch.nn.ModuleList()
    self.pools = torch.nn.ModuleList()
    self.convs.extend([GraphConv(hidden_features, hidden_features, aggr='mean') for i in range(num_layers - 1)])
    self.pools.extend([SAGPooling(hidden_features, ratio, GNN=GraphConv, min_score=min_score) for i in range((num_layers) // 2)])
    self.jump = JumpingKnowledge(mode='cat')
    self.lin1 = Linear(num_layers * hidden_features, hidden_features)
    self.lin2 = Linear(hidden_features, out_features)
    self.out_activation = out_activation
    self.dropout = dropout
self.dropout = dropout
```
In the above code, we begin by defining a single convolutional graph layer and then add two module list layers which allow us to pass in a variable number of layers. We then take our empty module list and append a variable number of GraphConv layers followed by a variable number of SAGPooling layers. We finish up our SAGPool definition by adding a JumpingKnowledge Layer, two linear layers, our activation function, and our dropout value. PyTorch’s intuitive syntax allows us to abstract away the complexity of working with state of the art methods like SAG Poolings while also maintaining the common approach to model development we are familiar with.
Improving Patient Outcomes
In order to achieve our mission of improving patient outcomes with AI-powered pathology, PathAI needs to rely on an ML development framework that (1) facilitates quick iteration and easy extension (i.e. Model configuration as code) during initial phases of development and exploration (2) scales model training and inference to massive images (3) easily and robustly serves models for production uses of our products (in clinical trials and beyond). As we’ve demonstrated, PyTorch offers us all of these capabilities and more. We are incredibly excited about the future of PyTorch and cannot wait to see what other impactful challenges we can solve using the framework.
Torch Distributed Elastic
Makes distributed PyTorch fault-tolerant and elastic.
Get Started
Usage
^^^^^


Quickstart


Train script


Examples


Documentation
API
^^^


torchrun (Elastic Launch)


Elastic Agent


Multiprocessing


Error Propagation


Rendezvous


Expiration Timers


Metrics


Events


Advanced
^^^^^^^^

Customization

Plugins
^^^^^^^

TorchElastic Kubernetes
torch.overrides
This module exposes various helper functions for the
"torch_function" protocol. See Extending torch for more detail on
the "torch_function" protocol.
Functions
torch.overrides.get_ignored_functions()
Return public functions that cannot be overridden by
   "torch_function".
Returns:
      A tuple of functions that are publicly available in the torch
      API but cannot be overridden with "torch_function". Mostly
      this is because none of the arguments of these functions are
      tensors or tensor-likes.
Return type:
      Set[Callable]
-[ Examples ]-



torch.Tensor.as_subclass in torch.overrides.get_ignored_functions()
   True
torch.add in torch.overrides.get_ignored_functions()
   False



torch.overrides.get_overridable_functions()
List functions that are overridable via torch_function
Returns:
      A dictionary that maps namespaces that contain overridable
functions to functions in that namespace that can be overridden.
Return type:
      Dict[Any, List[Callable]]
torch.overrides.resolve_name(f)
Get a human readable string name for a function passed to
   torch_function
Parameters:
      f (Callable) -- Function to resolve the name of.
Returns:
      Name of the function; if eval'ed it should give back the input
      function.
Return type:
      str
torch.overrides.get_testing_overrides()
Return a dict containing dummy overrides for all overridable
   functions
Returns:
      A dictionary that maps overridable functions in the PyTorch API
      to lambda functions that have the same signature as the real
      function and unconditionally return -1. These lambda functions
      are useful for testing API coverage for a type that defines
      "torch_function".
Return type:
      Dict[Callable, Callable]
-[ Examples ]-



import inspect
-[ Examples ]-



import inspect
my_add = torch.overrides.get_testing_overrides()[torch.add]
inspect.signature(my_add)
   



torch.overrides.handle_torch_function(public_api, relevant_args, args, *kwargs)
Implement a function with checks for "torch_function"
   overrides.
See torch::autograd::handle_torch_function for the equivalent of
   this function in the C++ implementation.
Parameters:
      * public_api (function) -- Function exposed by the public
        torch API originally called like "public_api(args, *kwargs)"
        on which arguments are now being checked.
  * **relevant_args** (*iterable*) -- Iterable of arguments to
    check for __torch_function__ methods.

  * **args** (*tuple*) -- Arbitrary positional arguments
    originally passed into "public_api".

  * **kwargs** (*tuple*) -- Arbitrary keyword arguments originally
    passed into "public_api".

Returns:
passed into "public_api".
Returns:
      Result from calling "implementation" or an "torch_function"
      method, as appropriate.
Return type:
      object
:raises TypeError : if no implementation is found.:
-[ Example ]-



def func(a):
   ...     if has_torch_function_unary(a):
   ...         return handle_torch_function(func, (a,), a)
   ...     return a + 0



torch.overrides.has_torch_function()
Check for torch_function implementations in the elements of an
   iterable or if a torch_function mode is enabled.  Considers
   exact "Tensor" s and "Parameter" s non-dispatchable.  Use this to
   guard a call to "handle_torch_function()"; don't use it to test if
   something is Tensor-like, use "is_tensor_like()" instead. :param
   relevant_args: Iterable or arguments to check for
   torch_function methods. :type relevant_args: iterable
Returns:
      True if any of the elements of relevant_args have
      torch_function implementations, False otherwise.
Return type:
      bool
See also:
 "torch.is_tensor_like"
    Checks if something is a Tensor-like, including an exact
    "Tensor".

torch.overrides.is_tensor_like(inp)
Returns "True" if the passed-in input is a Tensor-like.
Currently, this occurs whenever there's a "torch_function"
   attribute on the type of the input.
-[ Examples ]-
A subclass of tensor is generally a Tensor-like.



class SubTensor(torch.Tensor): ...
is_tensor_like(SubTensor([0]))
   True



Built-in or user types aren't usually Tensor-like.



is_tensor_like(6)
   False
is_tensor_like(None)
   False
class NotATensor: ...
is_tensor_like(NotATensor())
   False



But, they can be made Tensor-like by implementing
   torch_function.



class TensorLike:
   ...     @classmethod
   ...     def torch_function(cls, func, types, args, kwargs):
   ...         return -1
is_tensor_like(TensorLike())
   True
is_tensor_like(TensorLike())
   True



torch.overrides.is_tensor_method_or_property(func)
Returns True if the function passed in is a handler for a method or
   property belonging to "torch.Tensor", as passed into
   "torch_function".
Note:
 For properties, their "__get__" method must be passed in.

This may be needed, in particular, for the following reasons:


Methods/properties sometimes don't contain a module slot.


They require that the first passed-in argument is an instance of
      "torch.Tensor".


-[ Examples ]-



is_tensor_method_or_property(torch.Tensor.add)
   True
is_tensor_method_or_property(torch.add)
   False



Return type:
      bool
torch.overrides.wrap_torch_function(dispatcher)
Wraps a given function with "torch_function" -related
   functionality.
Parameters:
      dispatcher (Callable) -- A callable that returns an
      iterable of Tensor-likes passed into the function.
Note:
Note:
 This decorator may reduce the performance of your code.
 Generally, it's enough to express your code as a series of
 functions that, themselves, support __torch_function__. If you
 find yourself in the rare situation where this is not the case,
 e.g. if you're wrapping a low-level library and you also need it
 to work for Tensor-likes, then this function is available.

-[ Examples ]-



def dispatcher(a): # Must have the same signature as func
   ...     return (a,)
@torch.overrides.wrap_torch_function(dispatcher)
def func(a): # This will make func dispatchable by torch_function
   ...     return a + 0
Quantization Accuracy Debugging
This document provides high level strategies for improving
quantization accuracy. If a quantized model has error compared to the
original model, we can categorize the error into:


data insensitive error - caused by intrinsic model quantization
   error, large portion of input data has large error


data sensitive error - caused by outlier input data, small
   portion of input data has large error


implementation error - quantized kernel is not matching
   reference implementation


Data insensitive error
General tips

For PTQ, ensure that the data you are calibrating with is
   representative of your dataset. For example, for a classification
   problem a general guideline is to have multiple samples in every
   category, and the overall number of samples should be at least 100.
   There is no penalty for calibrating with more data other than
   calibration time.
calibration time.


If your model has Conv-BN or Linear-BN patterns, consider fusing
   them. If you are using FX graph mode quantization, this is done
   automatically by the workflow. If you are using Eager mode
   quantization, you can do this manually with the
   "torch.ao.quantization.fuse_modules" API.


Increase the precision of dtype of the problematic ops. Usually,
   fp32 will have the highest accuracy, followed by fp16, followed by
   dynamically quantized int8, followed by statically quantized int8.


Note: this is trading off performance for accuracy.


Note: availability of kernels per dtype per op can vary by
      backend.


Note: dtype conversions add an additional performance cost. For
      example, "fp32_op -> quant -> int8_op -> dequant -> fp32_op ->
      quant -> int8_op -> dequant" will have a performance penalty
      compared to "fp32_op -> fp32_op -> quant -> int8_op -> int8_op
      -> dequant" because of a higher number of required dtype
      conversions.
conversions.

If you are using PTQ, consider using QAT to recover some of the
   accuracy loss from quantization.

Int8 quantization tips


If you are using per-tensor weight quantization, consider using
   per-channel weight quantization.


If you are doing inference on fbgemm, ensure that you set the
   reduce_range argument to False if your CPU is Cooperlake or
   newer, and to True otherwise.


Audit the input activation distribution variation across different
   samples. If this variation is high, the layer may be suitable for
   dynamic quantization but not static quantization.


Data sensitive error
If you are using static quantization and a small portion of your input
data is resulting in high quantization error, you can try:


Adjust your calibration dataset to make it more representative of
   your inference dataset.


Manually inspect (using Numeric Suite) which layers have high
quantization error. For these layers, consider leaving them in
   floating point or adjusting the observer settings to choose a
   better scale and zero_point.
Implementation error
If you are using PyTorch quantization with your own backend you may
see differences between the reference implementation of an operation
(such as "dequant -> op_fp32 -> quant") and the quantized
implementation (such as op_int8) of the op on the target hardware.
This could mean one of two things:


the differences (usually small) are expected due to specific
   behavior of the target kernel on the target hardware compared to
   fp32/cpu. An example of this is accumulating in an integer dtype.
   Unless the kernel guarantees bitwise equivalency with the reference
   implementation, this is expected.


the kernel on the target hardware has an accuracy issue. In this
   case, reach out to the kernel developer.


Numerical Debugging Tooling (prototype)
Warning:
Distributed Optimizers
Warning:
Distributed optimizer is not currently supported when using CUDA
  tensors
"torch.distributed.optim" exposes DistributedOptimizer, which takes a
list of remote parameters ("RRef") and runs the optimizer locally on
the workers where the parameters live.  The distributed optimizer can
use any of the local optimizer Base class to apply the gradients on
each worker.
class torch.distributed.optim.DistributedOptimizer(optimizer_class, params_rref, args, *kwargs)
DistributedOptimizer takes remote references to parameters
   scattered across workers and applies the given optimizer locally
   for each parameter.
This class uses "get_gradients()" in order to retrieve the
   gradients for specific parameters.
Concurrent calls to "step()", either from the same or different
   clients, will be serialized on each worker -- as each worker's
   optimizer can only work on one set of gradients at a time. However,
there is no guarantee that the full forward-backward-optimizer
   sequence will execute for one client at a time. This means that the
   gradients being applied may not correspond to the latest forward
   pass executed on a given worker. Also, there is no guaranteed
   ordering across workers.
DistributedOptimizer creates the local optimizer with TorchScript
   enabled by default, so that optimizer updates are not blocked by
   the Python Global Interpreter Lock (GIL) in the case of
   multithreaded training (e.g. Distributed Model Parallel). This
   feature is currently enabled for most optimizers. You can also
   follow the recipe in PyTorch tutorials to enable TorchScript
   support for your own custom optimizers.
Parameters:
      * optimizer_class (optim.Optimizer) -- the class of
        optimizer to instantiate on each worker.
  * **params_rref** (*list**[**RRef**]*) -- list of RRefs to local
    or remote parameters to optimize.
or remote parameters to optimize.
  * **args** -- arguments to pass to the optimizer constructor on
    each worker.

  * **kwargs** -- arguments to pass to the optimizer constructor
    on each worker.

Example::
      >>> import torch.distributed.autograd as dist_autograd
      >>> import torch.distributed.rpc as rpc
      >>> from torch import optim
      >>> from torch.distributed.optim import DistributedOptimizer
      >>>
      >>> with dist_autograd.context() as context_id:
      >>>   # Forward pass.
      >>>   rref1 = rpc.remote("worker1", torch.add, args=(torch.ones(2), 3))
      >>>   rref2 = rpc.remote("worker1", torch.add, args=(torch.ones(2), 1))
      >>>   loss = rref1.to_here() + rref2.to_here()
      >>>
      >>>   # Backward pass.
      >>>   dist_autograd.backward(context_id, [loss.sum()])
      >>>
      >>>   # Optimizer.
      >>>   dist_optim = DistributedOptimizer(
      >>>      optim.SGD,
      >>>      [rref1, rref2],
      >>>      lr=0.05,
lr=0.05,
  >>>   )
  >>>   dist_optim.step(context_id)




step(context_id)
  Performs a single optimization step.

  This will call "torch.optim.Optimizer.step()" on each worker
  containing parameters to be optimized, and will block until all
  workers return. The provided "context_id" will be used to
  retrieve the corresponding "context" that contains the gradients
  that should be applied to the parameters.

  Parameters:
     **context_id** -- the autograd context id for which we should
     run the optimizer step.

class torch.distributed.optim.PostLocalSGDOptimizer(optim, averager)
Wraps an arbitrary "torch.optim.Optimizer" and runs post-local SGD,
   This optimizer runs local optimizer at every step. After the warm-
   up stage, it averages parameters periodically afer the local
   optimizer is applied.
Parameters:
      * optim (Optimizer) -- The local optimizer.