shrinath-suresh/blogs-docs-splitted huggingface dataset:
High level API: Extend mobile native APIs to cover common preprocessing and integration tasks needed for incorporating ML in mobile applications. e.g. Computer vision and NLP

Learn more or get started on Android or iOS here.
New tools for model interpretability and privacy
Captum
As models become ever more complex, it is increasingly important to develop new methods for model interpretability. To help address this need, we’re launching Captum, a tool to help developers working in PyTorch understand why their model generates a specific output. Captum provides state-of-the-art tools to understand how the importance of specific neurons and layers and affect predictions made by the models. Captum’s algorithms include integrated gradients, conductance, SmoothGrad and VarGrad, and DeepLift.
The example below shows how to apply model interpretability algorithms on a pretrained ResNet model and then visualize the attributions for each pixel by overlaying them on the image.
noise_tunnel = NoiseTunnel(integrated_gradients)

attributions_ig_nt, delta = noise_tunnel.attribute(input, n_samples=10, nt_type='smoothgrad_sq', target=pred_label_idx)
_ = viz.visualize_image_attr_multiple(["original_image", "heat_map"],
                                      ["all", "positive"],
                                      np.transpose(attributions_ig_nt.squeeze().cpu().detach().numpy(), (1,2,0)),
                                      np.transpose(transformed_img.squeeze().cpu().detach().numpy(), (1,2,0)),
                                      cmap=default_cmap,
                                      show_colorbar=True)







Learn more about Captum at captum.ai.
CrypTen
CrypTen
Practical applications of ML via cloud-based or machine-learning-as-a-service (MLaaS) platforms pose a range of security and privacy challenges. In particular, users of these platforms may not want or be able to share unencrypted data, which prevents them from taking full advantage of ML tools. To address these challenges, the ML community is exploring a number of technical approaches, at various levels of maturity. These include homomorphic encryption, secure multiparty computation, trusted execution environments, on-device computation, and differential privacy.
To provide a better understanding of how some of these technologies can be applied, we are releasing CrypTen, a new community-based research platform for taking the field of privacy-preserving ML forward. Learn more about CrypTen here. It is available on GitHub here.
Tools for multimodal AI systems
Digital content is often made up of several modalities, such as text, images, audio, and video. For example, a single public post might contain an image, body text, a title, a video, and a landing page. Even one particular component may have more than one modality, such as a video that contains both visual and audio signals, or a landing page that is composed of images, text, and HTML sources.
The ecosystem of tools and libraries that work with PyTorch offer enhanced ways to address the challenges of building multimodal ML systems. Here are some of the latest libraries launching today:
Detectron2
Detectron2
Object detection and segmentation are used for tasks ranging from autonomous vehicles to content understanding for platform integrity. To advance this work, Facebook AI Research (FAIR) is releasing Detectron2, an object detection library now implemented in PyTorch. Detectron2 provides support for the latest models and tasks, increased flexibility to aid computer vision research, and improvements in maintainability and scalability to support production use cases.
Detectron2 is available here and you can learn more here.
Speech extensions to fairseq
Speech extensions to fairseq
Language translation and audio processing are critical components in systems and applications such as search, translation, speech, and assistants. There has been tremendous progress in these fields recently thanks to the development of new architectures like transformers, as well as large-scale pretraining methods. We’ve extended fairseq, a framework for sequence-to-sequence applications such as language translation, to include support for end-to-end learning for speech and audio recognition tasks.These extensions to fairseq enable faster exploration and prototyping of new speech research ideas while offering a clear path to production.
Get started with fairseq here.
Cloud provider and hardware ecosystem support
Google Cloud TPU support now broadly available. To accelerate the largest-scale machine learning (ML) applications deployed today and enable rapid development of the ML applications of tomorrow, Google created custom silicon chips called Tensor Processing Units (TPUs). When assembled into multi-rack ML supercomputers called Cloud TPU Pods, these TPUs can complete ML workloads in minutes or hours that previously took days or weeks on other systems. Engineers from Facebook, Google, and Salesforce worked together to enable and pilot Cloud TPU support in PyTorch, including experimental support for Cloud TPU Pods. PyTorch support for Cloud TPUs is also available in Colab. Learn more about how to get started with PyTorch on Cloud TPUs here.
Alibaba adds support for PyTorch in Alibaba Cloud. The initial integration involves a one-click solution for PyTorch 1.x, Data Science Workshop notebook service, distributed training with Gloo/NCCL, as well as seamless integration with Alibaba IaaS such as OSS, ODPS, and NAS. Together with the toolchain provided by Alibaba, we look forward to significantly reducing the overhead necessary for adoption, as well as helping Alibaba Cloud’s global customer base leverage PyTorch to develop new AI applications.
ML hardware ecosystem expands. In addition to key GPU and CPU partners, the PyTorch ecosystem has also enabled support for dedicated ML accelerators. Updates from Intel and Habana showcase how PyTorch, connected to the Glow optimizing compiler, enables developers to utilize these market-specific solutions.

Growth in the PyTorch community
Growth in the PyTorch community
As an open source, community-driven project, PyTorch benefits from wide range of contributors bringing new capabilities to the ecosystem. Here are some recent examples:

Mila SpeechBrain aims to provide an open source, all-in-one speech toolkit based on PyTorch. The goal is to develop a single, flexible, user-friendly toolkit that can be used to easily develop state-of-the-art systems for speech recognition (both end to end and HMM-DNN), speaker recognition, speech separation, multi-microphone signal processing (e.g., beamforming), self-supervised learning, and many others. Learn more
SpaCy is a new wrapping library with consistent and easy-to-use interfaces to several models, in order to extract features to power NLP pipelines. Support is provided for via spaCy’s standard training API. The library also calculates an alignment so the transformer features can be related back to actual words instead of just wordpieces. Learn more
HuggingFace PyTorch-Transformers (formerly known as pytorch-pretrained-bert is a library of state-of-the-art pretrained models for Natural Language Processing (NLP). The library currently contains PyTorch implementations, pretrained model weights, usage scripts, and conversion utilities for models such as BERT, GPT-2, RoBERTa, and DistilBERT. It has also grown quickly, with more than 13,000 GitHub stars and a broad set of users. Learn more
PyTorch Lightning is a Keras-like ML library for PyTorch. It leaves core training and validation logic to you and automates the rest. Reproducibility is a crucial requirement for many fields of research, including those based on ML techniques. As the number of research papers submitted to arXiv and conferences skyrockets into the tens of thousands, scaling reproducibility becomes difficult. Learn more.

We recently held the first online Global PyTorch Summer Hackathon, where researchers and developers around the world were invited to build innovative new projects with PyTorch. Nearly 1,500 developers participated, submitting projects ranging from livestock disease detection to AI-powered financial assistants. The winning projects were:
Torchmeta, which provides extensions for PyTorch to simplify the development of meta-learning algorithms in PyTorch. It features a unified interface inspired by TorchVision for both few-shot classification and regression problems, to allow easy benchmarking on multiple data sets to aid with reproducibility.
Open-Unmix, a system for end-to-end music demixing with PyTorch. Demixing separates the individual instruments or vocal track from any stereo recording.
Endless AI-Generated Tees, a store featuring AI-generated T-shirt designs that can be purchased and delivered worldwide. The system uses a state-of-the-art generative model (StyleGAN) that was built with PyTorch and then trained on modern art.

Visit pytorch.org to learn more and get started with PyTorch 1.3 and the latest libraries and ecosystem projects. We look forward to the contributions, exciting research advancements, and real-world applications that the community builds with PyTorch.
layout: blog_detail
title: "New library updates in PyTorch 1.12"
author: Team PyTorch
featured-img: ''

We are bringing a number of improvements to the current PyTorch libraries, alongside the PyTorch 1.12 release. These updates demonstrate our focus on developing common and extensible APIs across all domains to make it easier for our community to build ecosystem projects on PyTorch. 
Summary:
- TorchVision - Added multi-weight support API, new architectures, model variants, and pretrained weight. See the release notes here.
- TorchAudio - Introduced beta features including a streaming API, a CTC beam search decoder, and new beamforming modules and methods. See the release notes here.
TorchText - Extended support for scriptable BERT tokenizer and added datasets for GLUE benchmark. See the release notes here.
TorchRec - Added EmbeddingModule benchmarks, examples for TwoTower Retrieval, inference and sequential embeddings, metrics, improved planner and demonstrated integration with production components. See the release notes here.
TorchX - Launch PyTorch trainers developed on local workspaces onto five different types of schedulers. See the release notes here.
FBGemm - Added and improved kernels for Recommendation Systems inference workloads, including table batched embedding bag, jagged tensor operations, and other special-case optimizations.

TorchVision v0.13
Multi-weight support API
Multi-weight support API
TorchVision v0.13 offers a new Multi-weight support API for loading different weights to the existing model builder methods:
from torchvision.models import *

# Old weights with accuracy 76.130%
resnet50(weights=ResNet50_Weights.IMAGENET1K_V1)

# New weights with accuracy 80.858%
resnet50(weights=ResNet50_Weights.IMAGENET1K_V2)

# Best available weights (currently alias for IMAGENET1K_V2)
# Note that these weights may change across versions
resnet50(weights=ResNet50_Weights.DEFAULT)

# Strings are also supported
resnet50(weights="IMAGENET1K_V2")

# No weights - random initialization
resnet50(weights=None)

The new API bundles along with the weights important details such as the preprocessing transforms and meta-data such as labels. Here is how to make the most out of it:
```python
from torchvision.io import read_image
from torchvision.models import resnet50, ResNet50_Weights
img = read_image("test/assets/encode_jpeg/grace_hopper_517x606.jpg")
Step 1: Initialize model with the best available weights
weights = ResNet50_Weights.DEFAULT
model = resnet50(weights=weights)
model.eval()
Step 2: Initialize the inference transforms
preprocess = weights.transforms()
Step 3: Apply inference preprocessing transforms
batch = preprocess(img).unsqueeze(0)
Step 4: Use the model and print the predicted category
prediction = model(batch).squeeze(0).softmax(0)
class_id = prediction.argmax().item()
score = prediction[class_id].item()
category_name = weights.meta["categories"][class_id]
print(f"{category_name}: {100 * score:.1f}%")
```
You can read more about the new API in the docs. To provide your feedback, please use this dedicated Github issue.
New architectures and model variants
Classification
Classification
The Swin Transformer and EfficienetNetV2 are two popular classification models which are often used for downstream vision tasks. This release includes 6 pre-trained weights for their classification variants. Here is how to use the new models:
import torch
from torchvision.models import *

image = torch.rand(1, 3, 224, 224)
model = swin_t(weights="DEFAULT").eval()
prediction = model(image)

image = torch.rand(1, 3, 384, 384)
model = efficientnet_v2_s(weights="DEFAULT").eval()
prediction = model(image)

In addition to the above, we also provide new variants for existing architectures such as ShuffleNetV2, ResNeXt and MNASNet. The accuracies of all the new pre-trained models obtained on ImageNet-1K are seen below:



Model
Acc@1
Acc@5




swin_t
81.474
95.776
| swin_s                         |    83.196 |     96.36 |
| swin_b                         |    83.582 |     96.64 |
| efficientnet_v2_s              |    84.228 |    96.878 |
| efficientnet_v2_m              |    85.112 |    97.156 |
| efficientnet_v2_l              |    85.808 |    97.788 |
| resnext101_64x4d               |    83.246 |    96.454 |
| resnext101_64x4d (quantized)   |    82.898 |    96.326 |
| shufflenet_v2_x1_5             |    72.996 |    91.086 |
| shufflenet_v2_x1_5 (quantized) |    72.052 |     0.700 |
| shufflenet_v2_x2_0             |    76.230 |    93.006 |
| shufflenet_v2_x2_0 (quantized) |    75.354 |    92.488 |
| mnasnet0_75                    |    71.180 |    90.496 |
| mnas1_3                        |    76.506 |    93.522 |
We would like to thank Hu Ye for contributing to TorchVision the Swin Transformer implementation.
(BETA) Object Detection and Instance Segmentation
We have introduced 3 new model variants for RetinaNet, FasterRCNN and MaskRCNN that include several post-paper architectural optimizations and improved training recipes. All models can be used similarly:
import torch
from torchvision.models.detection import *

images = [torch.rand(3, 800, 600)]
model = retinanet_resnet50_fpn_v2(weights="DEFAULT")
# model = fasterrcnn_resnet50_fpn_v2(weights="DEFAULT")
# model = maskrcnn_resnet50_fpn_v2(weights="DEFAULT")
model.eval()
prediction = model(images)

Below we present the metrics of the new variants on COCO val2017. In parenthesis we denote the improvement over the old variants:



Model
Box mAP
Mask mAP




retinanet_resnet50_fpn_v2
41.5 (+5.1)
-


fasterrcnn_resnet50_fpn_v2
46.7 (+9.7)
-


maskrcnn_resnet50_fpn_v2
47.4 (+9.5)
41.8 (+7.2)
We would like to thank Ross Girshick, Piotr Dollar, Vaibhav Aggarwal, Francisco Massa and Hu Ye for their past research and contributions to this work.
New pre-trained weights
SWAG weights
The ViT and RegNet model variants offer new pre-trained SWAG (​​Supervised Weakly from hashtAGs) weights. One of the biggest of these models achieves a whopping 88.6% accuracy on ImageNet-1K. We currently offer two versions of the weights: 1) fine-tuned end-to-end weights on ImageNet-1K (highest accuracy) and 2) frozen trunk weights with a linear classifier fit on ImageNet-1K (great for transfer learning). Below we see the detailed accuracies of each model variant:



Model Weights
Acc@1
Acc@5




RegNet_Y_16GF_Weights.IMAGENET1K_SWAG_E2E_V1
86.012
98.054


RegNet_Y_16GF_Weights.IMAGENET1K_SWAG_LINEAR_V1
83.976
97.244
| RegNet_Y_32GF_Weights.IMAGENET1K_SWAG_E2E_V1     |    86.838 |    98.362 |
| RegNet_Y_32GF_Weights.IMAGENET1K_SWAG_LINEAR_V1  |    84.622 |     97.48 |
| RegNet_Y_128GF_Weights.IMAGENET1K_SWAG_E2E_V1    |    88.228 |    98.682 |
| RegNet_Y_128GF_Weights.IMAGENET1K_SWAG_LINEAR_V1 |    86.068 |    97.844 |
| ViT_B_16_Weights.IMAGENET1K_SWAG_E2E_V1          |    85.304 |     97.65 |
| ViT_B_16_Weights.IMAGENET1K_SWAG_LINEAR_V1       |    81.886 |     96.18 |
| ViT_L_16_Weights.IMAGENET1K_SWAG_E2E_V1          |    88.064 |    98.512 |
| ViT_L_16_Weights.IMAGENET1K_SWAG_LINEAR_V1       |    85.146 |    97.422 |
| ViT_H_14_Weights.IMAGENET1K_SWAG_E2E_V1          |    88.552 |    98.694 |
| ViT_H_14_Weights.IMAGENET1K_SWAG_LINEAR_V1       |    85.708 |     97.73 |
The SWAG weights are released under the Attribution-NonCommercial 4.0 International license. We would like to thank Laura Gustafson, Mannat Singh and Aaron Adcock for their work and support in making the weights available to TorchVision.
Model Refresh
The release of the Multi-weight support API enabled us to refresh the most popular models and offer more accurate weights. We improved on average each model by ~3 points. The new recipe used was learned on top of ResNet50 and its details were covered on a previous blog post. 



Model
Old weights
New weights




efficientnet_b1
78.642
79.838


mobilenet_v2
71.878
72.154
| mobilenet_v3_large           |          74.042 |          75.274 |
| regnet_y_400mf               |          74.046 |          75.804 |
| regnet_y_800mf               |           76.42 |          78.828 |
| regnet_y_1_6gf               |           77.95 |          80.876 |
| regnet_y_3_2gf               |          78.948 |          81.982 |
| regnet_y_8gf                 |          80.032 |          82.828 |
| regnet_y_16gf                |          80.424 |          82.886 |
| regnet_y_32gf                |          80.878 |          83.368 |
| regnet_x_400mf               |          72.834 |          74.864 |
| regnet_x_800mf               |          75.212 |          77.522 |
| regnet_x_1_6gf               |           77.04 |          79.668 |
| regnet_x_3_2gf               |          78.364 |          81.196 |
| regnet_x_8gf                 |          79.344 |          81.682 |
| regnet_x_16gf                |          80.058 |          82.716 |
| regnet_x_32gf                |          80.622 |          83.014 |
| resnet50                     |           76.13 |          80.858 |
| resnet50 (quantized)         |           75.92 |          80.282 |
| resnet101                    |          77.374 |          81.886 |
| resnet152                    |          78.312 |          82.284 |
| resnext50_32x4d              |          77.618 |          81.198 |
| resnext101_32x8d             |          79.312 |          82.834 |
| resnext101_32x8d (quantized) |          78.986 |          82.574 |
| wide_resnet50_2              |          78.468 |          81.602 |
| wide_resnet101_2             |          78.848 |           82.51 |
We would like to thank Piotr Dollar, Mannat Singh and Hugo Touvron for their past research and contributions to this work.
New Augmentations, Layers and Losses
New Augmentations, Layers and Losses
This release brings a bunch of new primitives which can be used to produce SOTA models. Some highlights include the addition of AugMix data-augmentation method, the DropBlock layer, the cIoU/dIoU loss and many more. We would like to thank Aditya Oke, Abhijit Deo, Yassine Alouini and Hu Ye for contributing to the project and for helping us maintain TorchVision relevant and fresh.
Documentation
We completely revamped our models documentation to make them easier to browse, and added various key information such as supported image sizes, or image pre-processing steps of pre-trained weights. We now have a main model page with various summary tables of available weights, and each model has a dedicated page. Each model builder is also documented in their own page, with more details about the available weights, including accuracy, minimal image size, link to training recipes, and other valuable info. For comparison, our previous models docs are here. To provide feedback on the new documentation, please use the dedicated Github issue.
TorchAudio v0.12
(BETA) Streaming API

 

StreamReader is TorchAudio’s new I/O API. It is backed by FFmpeg†, and allows users to:
- Decode audio and video formats, including MP4 and AAC
- Handle input forms, such as local files, network protocols, microphones, webcams, screen captures and file-like objects
- Iterate over and decode chunk-by-chunk, while changing the sample rate or frame rate
- Apply audio and video filters, such as low-pass filter and image scaling
- Decode video with Nvidia's hardware-based decoder (NVDEC)
For usage details, please check out the documentation and tutorials:
- Media Stream API - Pt.1
Media Stream API - Pt.2
Online ASR with Emformer RNN-T
Device ASR with Emformer RNN-T
Accelerated Video Decoding with NVDEC

† To use StreamReader, FFmpeg libraries are required. Please install FFmpeg. The coverage of codecs depends on how these libraries are configured. TorchAudio official binaries are compiled to work with FFmpeg 4 libraries; FFmpeg 5 can be used if TorchAudio is built from source.
(BETA) CTC Beam Search Decoder
TorchAudio integrates the wav2letter CTC beam search decoder from Flashlight (GitHub). The addition of this inference time decoder enables running end-to-end CTC ASR evaluation using TorchAudio utils.
Customizable lexicon and lexicon-free decoders are supported, and both are compatible with KenLM n-gram language models or without using a language model. TorchAudio additionally supports downloading token, lexicon, and pretrained KenLM files for the LibriSpeech dataset.
For usage details, please check out the documentation and ASR inference tutorial.
(BETA) New Beamforming Modules and Methods
To improve flexibility in usage, the release adds two new beamforming modules under torchaudio.transforms: SoudenMVDR and RTFMVDR. The main differences from MVDR are:
Use power spectral density (PSD) and relative transfer function (RTF) matrices as inputs instead of time-frequency masks. The module can be integrated with neural networks that directly predict complex-valued STFT coefficients of speech and noise
Add \'reference_channel\' as an input argument in the forward method, to allow users to select the reference channel in model training or dynamically change the reference channel in inference

Besides the two modules, new function-level beamforming methods are added under torchaudio.functional. These include:
- psd
- mvdr_weights_souden
- mvdr_weights_rtf
- rtf_evd
- rtf_power
apply_beamforming

For usage details, please check out the documentation at torchaudio.transforms and torchaudio.functional and the Speech Enhancement with MVDR Beamforming tutorial.
TorchText v0.13
Glue Datasets
We increased the number of datasets in TorchText from 22 to 30 by adding the remaining 8 datasets from the GLUE benchmark (SST-2 was already supported). The complete list of GLUE datasets is as follows:
- CoLA (paper): Single sentence binary classification acceptability task
- SST-2 (paper): Single sentence binary classification sentiment task
WNLI (paper): Dual sentence binary classification coreference and NLI tasks

Scriptable BERT Tokenizer
TorchText has extended support for scriptable tokenizer by adding the WordPiece tokenizer used in BERT. It is one of the commonly used algorithms for splitting input text into sub-words units and was introduced in Japanese and Korean Voice Search (Schuster et al., 2012).  
TorchScriptabilty support would allow users to embed the BERT text-preprocessing natively in C++ without needing the support of python runtime. As TorchText now supports the CMAKE build system to natively link torchtext binaries with application code, users can easily integrate BERT tokenizers for deployment needs.
For usage details, please refer to the corresponding documentation.
TorchRec v0.2.0
EmbeddingModule + DLRM benchmarks
A set of benchmarking tests, showing performance characteristics of TorchRec’s base modules  and research models built out of TorchRec.
TwoTower Retrieval Example, with FAISS
We provide an example demonstrating training a distributed TwoTower (i.e. User-Item) Retrieval model that is sharded using TorchRec. The projected item embeddings are added to an IVFPQ FAISS index for candidate generation. The retrieval model and KNN lookup are bundled in a Pytorch model for efficient end-to-end retrieval.
Integrations
We demonstrate that TorchRec works out of the box with many components commonly used alongside PyTorch models in production like systems, such as