shrinath-suresh/blogs-docs-splitted huggingface dataset:
layout: blog_detail
title: 'PyTorch 1.10 Release, including CUDA Graphs APIs, Frontend and Compiler Improvements'
author: Team PyTorch

We are excited to announce the release of PyTorch 1.10. This release is composed of over 3,400 commits since 1.9, made by 426 contributors. We want to sincerely thank our community for continuously improving PyTorch. 
PyTorch 1.10 updates are focused on improving training and performance of PyTorch, and developer usability. The full release notes are available here. Highlights include:
1. CUDA Graphs APIs are integrated to reduce CPU overheads for CUDA workloads.
2. Several frontend APIs such as FX, torch.special, and nn.Module Parametrization, have moved from beta to stable.
3. Support for automatic fusion in JIT Compiler expands to CPUs in addition to GPUs.
4. Android NNAPI support is now available in beta.
Along with 1.10, we are also releasing major updates to the PyTorch libraries, which you can read about in this blog post.
Frontend APIs
(Stable) Python code transformations with FX
FX provides a Pythonic platform for transforming and lowering PyTorch programs. It is a toolkit for pass writers to facilitate Python-to-Python transformation of functions and nn.Module instances. This toolkit aims to support a subset of Python language semantics—rather than the whole Python language—to facilitate ease of implementation of transforms. With 1.10, FX is moving to stable. 
You can learn more about FX in the official documentation and GitHub examples of program transformations implemented using torch.fx.
(Stable) torch.special
(Stable) torch.special
A torch.special module, analogous to SciPy’s special module, is now available in stable. The module has 30 operations, including gamma, Bessel, and (Gauss) error functions. 
Refer to this documentation for more details.
(Stable) nn.Module Parametrization
nn.Module parametrizaton, a feature that allows users to parametrize any parameter or buffer of an nn.Module without modifying the nn.Module itself, is available in stable. This release adds weight normalization (weight_norm), orthogonal parametrization (matrix constraints and part of pruning) and more flexibility when creating your own parametrization.
Refer to this tutorial and the general documentation for more details.
(Beta) CUDA Graphs APIs Integration
PyTorch now integrates CUDA Graphs APIs to reduce CPU overheads for CUDA workloads.
CUDA Graphs greatly reduce the CPU overhead for CPU-bound cuda workloads and thus improve performance by increasing GPU utilization. For distributed workloads, CUDA Graphs also reduce jitter, and since parallel workloads have to wait for the slowest worker, reducing jitter improves overall parallel efficiency.
Integration allows seamless interop between the parts of the network captured by cuda graphs, and parts of the network that cannot be captured due to graph limitations.
Read the note for more details and examples, and refer to the general documentation for additional information. 
[Beta] Conjugate View
PyTorch’s conjugation for complex tensors (torch.conj()) is now a constant time operation, and returns a view of the input tensor with a conjugate bit set as can be seen by calling torch.is_conj() . This has already been leveraged in various other PyTorch operations like matrix multiplication, dot product etc., to fuse conjugation with the operation leading to significant performance gain and memory savings on both CPU and CUDA.
Distributed Training
Distributed Training Releases Now in Stable
Distributed Training Releases Now in Stable
In 1.10, there are a number of features that are moving from beta to stable in the distributed package:
* (Stable) Remote Module: This feature allows users to operate a module on a remote worker like using a local module, where the RPCs are transparent to the user. Refer to this documentation for more details.
* (Stable) DDP Communication Hook: This feature allows users to override how DDP synchronizes gradients across processes. Refer to this documentation for more details.
(Stable) ZeroRedundancyOptimizer: This feature can be used in conjunction with DistributedDataParallel to reduce the size of per-process optimizer states. With this stable release, it now can handle uneven inputs to different data-parallel workers. Check out this tutorial. We also improved the parameter partition algorithm to better balance memory and computation overhead across processes. Refer to this documentation and this tutorial to learn more. 

Performance Optimization and Tooling
[Beta] Profile-directed typing in TorchScript
[Beta] Profile-directed typing in TorchScript
TorchScript has a hard requirement for source code to have type annotations in order for compilation to be successful. For a long time, it was only possible to add missing or incorrect type annotations through trial and error (i.e., by fixing the type-checking errors generated by torch.jit.script one by one), which was inefficient and time consuming. 
Now, we have enabled profile directed typing for torch.jit.script by leveraging existing tools like MonkeyType, which makes the process much easier, faster, and more efficient. For more details, refer to the documentation.
(Beta) CPU Fusion
In PyTorch 1.10, we've added an LLVM-based JIT compiler for CPUs that can fuse together sequences of torch library calls to improve performance. While we've had this capability for some time on GPUs, this release is the first time we've brought compilation to the CPU.
You can check out a few performance results for yourself in this Colab notebook.
(Beta) PyTorch Profiler
The objective of PyTorch Profiler is to target the execution steps that are the most costly in time and/or memory, and visualize the workload distribution between GPUs and CPUs. PyTorch 1.10 includes the following key features:

Enhanced Memory View: This helps you understand your memory usage better. This tool will help you avoid Out of Memory errors by showing active memory allocations at various points of your program run.
Enhanced Automated Recommendations: This helps provide automated performance recommendations to help optimize your model. The tools recommend changes to batch size, TensorCore, memory reduction technologies, etc.
Enhanced Kernel View: Additional columns show grid and block sizes as well as shared memory usage and registers per thread.
Distributed Training: Gloo is now supported for distributed training jobs.
Correlate Operators in the Forward & Backward Pass: This helps map the operators found in the forward pass to the backward pass, and vice versa, in a trace view.
TensorCore: This tool shows the Tensor Core (TC) usage and provides recommendations for data scientists and framework developers.
NVTX: Support for NVTX markers was ported from the legacy autograd profiler.
Support for profiling on mobile devices: The PyTorch profiler now has better integration with TorchScript and mobile backends, enabling trace collection for mobile workloads.

Refer to this documentation for details. Check out this tutorial to learn how to get started with this feature. 
PyTorch Mobile
(Beta) Android NNAPI Support in Beta
(Beta) Android NNAPI Support in Beta
Last year we released prototype support for Android’s Neural Networks API (NNAPI). NNAPI allows Android apps to run computationally intensive neural networks on the most powerful and efficient parts of the chips that power mobile phones, including GPUs (Graphics Processing Units) and NPUs (specialized Neural Processing Units). 
Since the prototype we’ve added more op coverage, added support for load-time flexible shapes and ability to run the model on the host for testing. Try out this feature using the tutorial.
Additionally, Transfer Learning steps have been added to Object Detection examples. Check out this GitHub page to learn more. Please provide your feedback or ask questions on the forum. You can also check out this presentation to get an overview. 
Thanks for reading. If you’re interested in these updates and want to join the PyTorch community, we encourage you to join the discussion forums and open GitHub issues. To get the latest news from PyTorch, follow us on Twitter, Medium, YouTube, and LinkedIn. 
Cheers!
Team PyTorch
layout: blog_detail
title: "Ambient Clinical Intelligence: Generating Medical Reports with PyTorch"
author: Miguel Del-Agua, Principal Research Scientist, Nuance and Jeremy Jancsary, Senior Principal Research Scientist, Nuance
featured-img: ""

Introduction
Complete and accurate clinical documentation is an essential tool for tracking patient care. It allows for treatment plans to be shared among care teams to aid in continuity of care and ensures a transparent and effective process for reimbursement.
Physicians are responsible for documenting patient care. Traditional clinical documentation methods have resulted in a sub-par patient-provider experience, less time interacting with patients, and decreased work-life balance.  A significant amount of physicians’ time is spent in front of the computer doing administrative tasks. As a result, patients are less satisfied with the overall experience, and physicians, who prepare for years studying medicine, cannot practice at the top of their license and are burned out. Every hour physicians provide direct clinical face time to patients results in nearly two additional hours spent on EHR and desk work within the clinic day. Outside office hours, physicians spend another 1 to 2 hours of personal time each night doing additional computer and other clerical work.

42% of all physicians reported having burnout. – Medscape
What can we do to bring back trust, joy, and humanity to the delivery of healthcare? A significant portion of the administrative work consists of entering patient data into Electronic Health Records (EHRs) and creating clinical documentation. Clinical documentation is created from information already in the EHR as well as from the patient-provider encounter conversation. 
This article will showcase how the Nuance Dragon Ambient eXperience (DAX), an AI-powered, voice-enabled, ambient clinical intelligence solution, automatically documents patient encounters accurately and efficiently at the point of care and the technologies that enable it.
Nuance DAX enhances the quality of care and patient experience, increases provider efficiency and satisfaction, and improves financial outcomes. It can be used in office and telehealth settings in all ambulatory specialties, including primary and urgent care.
Natural Language Processing
Natural Language Processing (NLP) is one of the most challenging fields in Artificial Intelligence (AI). It comprehends a set of algorithms that allow computers to understand or generate the language used by humans. These algorithms can process and analyze vast amounts of natural language data from different sources (either sound or text) to build models that can understand, classify, or even generate natural language as humans would. Like other fields in AI, NLP has significantly progressed thanks to the advent of Deep Learning (DL), which has resulted in models that can obtain results on par with humans in some tasks.
Two main NLP components play a role in automating the creation of clinical documentation. The first component, Automatic Speech Recognition (ASR), is used to translate speech into text. It takes the audio recording of the encounter and generates a conversation transcription (cf. Figure 2). The second component, Automatic Text Summarization, helps generate summaries from large text documents. This component is responsible for understanding and capturing the nuances and most essential aspects from the transcribed conversation into a final report in narrative form (cf. Figure 3), structured form, or a combination of both.
We will focus on this second component, Automatic Text Summarization, which is a difficult task with many challenges:

Its performance is tied to the ASR quality from multiple speakers (noisy input).
The input is conversational in nature and contains layman's terms.
Protected Health Information (PHI) regulations limit medical data access.
The information for one output sentence is potentially spread across multiple conversation turns.
There is no explicit sentence alignment between input and output.
Various medical specialties, encounter types, and EHR systems constitute a broad and complex output space. 
Physicians have different styles of conducting encounters and have their preferences for medical reports; there is no standard. 
Standard summarization metrics might differ from human judgment of quality.





Figure 2: Transcript of a patient-doctor conversation





Figure 3: Excerpt of an AI-generated medical report. HPI stands for History of present illness.

Text Summarization with PyTorch and Fairseq
Text Summarization with PyTorch and Fairseq
PyTorch is an open-source machine learning framework developed by Facebook that helps researchers prototype Deep Learning models. The Fairseq toolkit is built on top of PyTorch and focuses on sequence generation tasks, such as Neural Machine Translation (NMT) or Text Summarization. Fairseq features an active community that is continuously providing reference implementations of state-of-the-art models. It contains many built-in components (model architectures, modules, loss functions, and optimizers) and is easily extendable with plugins.
Text summarization constitutes a significant challenge in NLP. We need models capable of generating a short version of a document while retaining the key points and avoiding uninformative content. These  challenges can be addressed with  different approaches. 1). Abstractive text summarization aimed at training models that can generate a summary in narrative form. 2). Extractive methods where the models are trained to select the most important parts from the input text. 3). A combination of the two, where the essential parts from the input are selected and then summarized in an abstractive fashion. Hence, summarization can be accomplished via a single end-to-end network or as a pipeline of extractive and abstractive components. To that end, Fairseq provides all the necessary tools to be successful in our endeavor. It features either end-to-end models such as the classical Transformer, different types of Language Models and pre-trained versions that enable researchers to focus on what matters most—to build state-of-the-art models that generate valuable reports.
However, we are not just summarizing the transcribed conversation; we generate high-quality medical reports, which have many considerations.

Every section of a medical report is different in terms of content, structure, fluency, etc.
All medical facts mentioned in the conversation should be present in the report, for example, a particular treatment or dosage.
In the healthcare domain, the vocabulary is extensive, and models need to deal with medical terminology.
Patient-doctor conversations are usually much longer than the final report.
In our continuous effort to contribute to the open-source community, features have been developed at Nuance and pushed to the Fairseq GitHub repository.  These try to overcome some of the challenges mentioned such as, facilitating copying of, especially rare or unseen, words from the input to summary, training speedups by improving Tensor Core utilization, and ensuring TorchScript compatibility of different Transformer configurations. Following, we will show an example of how to train a Transformer model with a Pointer Generator mechanism (Transformer-PG), which can copy words from the input.
How to build a Transformer model with a Pointer Generator mechanism
In this step-by-step guide, it is assumed the user has already installed PyTorch and Fairseq.
1. Create a vocabulary and extend it with source position markers:
These markers will allow the model to point to any word in the input sequence.
```python
vocab_size=
position_markers=512
export LC_ALL=C
cat train.src train.tgt |
export LC_ALL=C
cat train.src train.tgt |
  tr -s '[:space:]' '\n' |
  sort |
  uniq -c |
  sort -k1,1bnr -k2 |
  head -n "$((vocab_size - 4))" |
  awk '{ print $2 " " $1 }' > dict.pg.txt
python3 -c "[print(' 0'.format(n)) for n in range($position_markers)]" >> dict.pg.txt

This will create a file "dict.pg.txt" that contains the \<vocab_size> most frequent words followed by 512 position markers named from "\<unk-0>" to "\<unk-511>".

In case we have an input like

```python
src = "Hello, I'm The Dogtor"

it could happen that our model has been trained without the word "Dogtor" in its vocabulary. Therefore, when we feed this sequence into the model, it should be converted to:
src = "Hello, I'm The <unk-3>"

Now, "\" is part of our vocabulary and could be predicted by the model (this is where the pointer-generator comes in). In such a case, we will only need to post-process the output to replace "\" by the word at input position 3.
2. Preprocess the text data to replace unknown words by its positional markers:
We can use the scripts from https://github.com/pytorch/fairseq/tree/master/examples/pointer_generator.
```python
Considering we have our data in:
train_src = /path/to/train.src
train_tgt = /path/to/train.tgt
valid_src = /path/to/valid.src
valid_tgt = /path/to/valid.tgt
./preprocess.py --source /path/to/train.src \
                --target /path/to/train.tgt \
                --vocab <(cut -d' ' -f1 dict.pg.txt) \
                --source-out /path/to/train.pg.src \
                --target-out /path/to/train.pg.tgt
./preprocess.py --source /path/to/valid.src \
                --target /path/to/valid.tgt \
                --vocab <(cut -d' ' -f1 dict.pg.txt) \
                --source-out /path/to/valid.pg.src \
                --target-out /path/to/valid.pg.tgt
./preprocess.py --source /path/to/test.src \
./preprocess.py --source /path/to/test.src \
                --vocab <(cut -d' ' -f1 dict.pg.txt) \
                --source-out /path/to/test.pg.src

### 3. Now let's binarize the data, so that it can be processed faster:

```python
fairseq-preprocess --task "translation" \
                   --source-lang "pg.src" \
                   --target-lang "pg.tgt" \
                   --trainpref /path/to/train \
                   --validpref /path/to/valid \
                   --srcdict dict.pg.txt \
                   --cpu \
                   --joined-dictionary \
                   --destdir <data_dir>

You might notice the type of task is "translation". This is because there is no "summarization" task available; we could understand it as a kind of NMT task where the input and output languages are shared and the output (summary) is shorter than the input.
4. Now we can train the model:
```python
fairseq-train  \
              --save-dir  \
--save-dir  \
              --task "translation" \
              --source-lang "src" \
              --target-lang "tgt" \
              --arch "transformer_pointer_generator" \
              --max-source-positions 512 \
              --max-target-positions 128 \
              --truncate-source \
              --max-tokens 2048 \
              --required-batch-size-multiple 1 \
              --required-seq-len-multiple 8 \
              --share-all-embeddings \
              --dropout 0.1 \
              --criterion "cross_entropy" \
              --optimizer adam \
              --adam-betas '(0.9, 0.98)' \
              --adam-eps 1e-9 \
              --update-freq 4 \
              --lr 0.004 \
              # Pointer Generator
              --alignment-layer -1 \
              --alignment-heads 1 \
              --source-position-markers 512
```
This configuration makes use of features Nuance has contributed back to Fairseq:
Transformer with a Pointer Generator mechanism to facilitate copying of words from the input.
Sequence length padded to a multiple of 8 to better use tensor cores and reduce training time.

5. Now let's take a look at how to generate a summary with our new medical report generation system:
```python
import torch
from examples.pointer_generator.pointer_generator_src.transformer_pg import TransformerPointerGeneratorModel
Patient-Doctor conversation
input = "[doctor] Lisa Simpson, thirty six year old female, presents to the clinic today because " \
        "she has severe right wrist pain"
Load the model
model = TransformerPointerGeneratorModel.from_pretrained(data_name_or_path=,
                                                         model_name_or_path=,
                                                         checkpoint_file="checkpoint_best.pt")
result = model.translate([input], beam=2)
print(result[0])
print(result[0])
Ms.  is a 36-year-old female who presents to the clinic today for evaluation of her right wrist.

### 6. Alternatively, we can use fairseq-interactive and a postprocessing tool to substitute positional unknown tokens by its words from the input:

```python
fairseq-interactive <data_dir> \
              --batch-size <batch_size> \
              --task translation \
              --source-lang src \
              --target-lang tgt \
              --path <model_dir>/checkpoint_last.pt \
              --input /path/to/test.pg.src \
              --buffer-size 20 \
              --max-len-a 0 \
              --max-len-b 128 \
              --beam 2 \
              --skip-invalid-size-inputs-valid-test | tee generate.out

grep "^H-" generate.out | cut -f 3- > generate.hyp

./postprocess.py \
    --source <(awk 'NF<512' /path/to/test.pg.src) \
    --target generate.hyp \
    --target-out generate.hyp.processed
--target-out generate.hyp.processed
```
Now we have the final set of reports in "generate.hyp.processed", with "\" replaced by the original word from the input sequence.
Model Deployment
PyTorch offers great flexibility in modeling and a rich surrounding ecosystem. However, while several recent articles have suggested that the use of PyTorch in research and academia may be close to surpassing TensorFlow, there seems to be an overall sense of TensorFlow being the preferred platform for deployment to production. Is this still the case in 2021? Teams looking to serve their PyTorch models in production have a few options.
Before describing our journey, let's take a brief detour and define the term model.
Models as computation graphs
A few years back, it was still common for machine learning toolkits to support only particular classes of models of a rather fixed and rigid structure, with only a few degrees of freedom (like the kernel of a support vector machine or the number of hidden layers of a neural network). Inspired by foundational work in Theano, toolkits like Microsoft's CNTK or Google's TensorFlow were among the first to popularize a more flexible view on models, as computation graphs with associated parameters that can be estimated from data. This view blurred the boundaries between popular types of models (such as DNNs or SVMs), as it became easy to blend the characteristics of each into your type of graph. Still, such a graph had to be defined upfront before estimating its parameters, and it was pretty static. This made it easy to save models to a self-contained bundle, like a TensorFlow SavedModel (such a bundle simply contains the structure of the graph, as well as the concrete values of the estimated parameters). However, debugging such models can be difficult because the statements in the Python code that build the graph are logically separate from the lines that execute it. Researchers also long for easier ways of expressing dynamic behavior, such as the computation steps of the forward pass of a model being conditionally dependent on its input data (or its previous output).