shrinath-suresh/blogs-docs-splitted huggingface dataset:
Although the most commonly encountered big data sets right now involve images and videos, big datasets occur in many other domains and involve many other kinds of data types: web pages, financial transactions, network traces, brain scans, etc.
However, working with the large amount of data sets presents a number of challenges:

Dataset Size: datasets often exceed the capacity of node-local disk storage, requiring distributed storage systems and efficient network access.
Number of Files: datasets often consist of billions of files with uniformly random access patterns, something that often overwhelms both local and network file systems.
Data Rates: training jobs on large datasets often use many GPUs, requiring aggregate I/O bandwidths to the dataset of many GBytes/s; these can only be satisfied by massively parallel I/O systems.
Shuffling and Augmentation: training data needs to be shuffled and augmented prior to training.
Scalability: users often want to develop and test on small datasets and then rapidly scale up to large datasets.

Traditional local and network file systems, and even object storage servers, are not designed for these kinds of applications. The WebDataset I/O library for PyTorch, together with the optional AIStore server and Tensorcom RDMA libraries, provide an efficient, simple, and standards-based solution to all these problems. The library is simple enough for day-to-day use, is based on mature open source standards, and is easy to migrate to from existing file-based datasets.
Using WebDataset is simple and requires little effort, and it will let you scale up the same code from running local experiments to using hundreds of GPUs on clusters or in the cloud with linearly scalable performance. Even on small problems and on your desktop, it can speed up I/O tenfold and simplifies data management and processing of large datasets. The rest of this blog post tells you how to get started with WebDataset and how it works.
The WebDataset Library
The WebDataset library provides a simple solution to the challenges listed above. Currently, it is available as a separate library (github.com/tmbdev/webdataset), but it is on track for being incorporated into PyTorch (see RFC 38419).  The WebDataset implementation is small (about 1500 LOC) and has no external dependencies.
Instead of inventing a new format, WebDataset represents large datasets as collections of POSIX tar archive files consisting of the original data files. The WebDataset library can use such tar archives directly for training, without the need for unpacking or local storage.
WebDataset scales perfectly from small, local datasets to petascale datasets and training on hundreds of GPUs and allows data to be stored on local disk, on web servers, or dedicated file servers. For container-based training, WebDataset eliminates the need for volume plugins or node-local storage. As an additional benefit, datasets need not be unpacked prior to training, simplifying the distribution and use of research data.
WebDataset implements PyTorch’s IterableDataset interface and can be used like existing DataLoader-based code. Since data is stored as files inside an archive, existing loading and data augmentation code usually requires minimal modification.
The WebDataset library is a complete solution for working with large datasets and distributed training in PyTorch (and also works with TensorFlow, Keras, and DALI via their Python APIs). Since POSIX tar archives are a standard, widely supported format, it is easy to write other tools for manipulating datasets in this format. E.g., the tarp command is written in Go and can shuffle and process training datasets.
Benefits
Benefits
The use of sharded, sequentially readable formats is essential for very large datasets. In addition, it has benefits in many other environments. WebDataset provides a solution that scales well from small problems on a desktop machine to very large deep learning problems in clusters or in the cloud. The following table summarizes some of the benefits in different environments.
{:.table.table-striped.table-bordered}
 | Environment  | Benefits of WebDataset |
| ------------- | ------------- |
| Local Cluster with AIStore | AIStore can be deployed easily as K8s containers and offers linear scalability and near 100% utilization of network and I/O bandwidth. Suitable for petascale deep learning. |
| Cloud Computing | WebDataset deep learning jobs can be trained directly against datasets stored in cloud buckets; no volume plugins required. Local and cloud jobs work identically. Suitable for petascale learning. |
| Local Cluster with existing distributed FS or object store |  WebDataset’s large sequential reads improve performance with existing distributed stores and eliminate the need for dedicated volume plugins. |
| Educational Environments | WebDatasets can be stored on existing web servers and web caches, and can be accessed directly by students by URL |
| Training on Workstations from Local Drives | Jobs can start training as the data still downloads. Data doesn’t need to be unpacked for training. Ten-fold improvements in I/O performance on hard drives over random access file-based datasets. |
| All Environments | Datasets are represented in an archival format and contain metadata such as file types. Data is compressed in native formats (JPEG, MP4, etc.). Data management, ETL-style jobs, and data transformations and I/O are simplified and easily parallelized.  |
We will be adding more examples giving benchmarks and showing how to use WebDataset in these environments over the coming months.
High-Performance
High-Performance
For high-performance computation on local clusters, the companion open-source AIStore server provides full disk to GPU I/O bandwidth, subject only to hardware constraints. This Bigdata 2019 Paper contains detailed benchmarks and performance measurements. In addition to benchmarks, research projects at NVIDIA and Microsoft have used WebDataset for petascale datasets and billions of training samples.
Below is a benchmark of AIStore with WebDataset clients using 12 server nodes with 10 rotational drives each.
The left axis shows the aggregate bandwidth from the cluster, while the right scale shows the measured per drive I/O bandwidth. WebDataset and AIStore scale linearly to about 300 clients, at which point they are increasingly limited by the maximum I/O bandwidth available from the rotational drives (about 150 MBytes/s per drive). For comparison, HDFS is shown. HDFS uses a similar approach to AIStore/WebDataset and also exhibits linear scaling up to about 192 clients; at that point, it hits a performance limit of about 120 MBytes/s per drive, and it failed when using more than 1024 clients. Unlike HDFS, the WebDataset-based code just uses standard URLs and HTTP to access data and works identically with local files, with files stored on web servers, and with AIStore. For comparison, NFS in similar experiments delivers about 10-20 MBytes/s per drive.
Storing Datasets in Tar Archives
Storing Datasets in Tar Archives
The format used for WebDataset is standard POSIX tar archives, the same archives used for backup and data distribution. In order to use the format to store training samples for deep learning, we adopt some simple naming conventions:
* datasets are POSIX tar archives
* each training sample consists of adjacent files with the same basename
* shards are numbered consecutively
For example, ImageNet is stored in 1282 separate 100 Mbyte shards with names pythonimagenet-train-000000.tar to imagenet-train-001281.tar, the contents of the first shard are:
```python
-r--r--r-- bigdata/bigdata      3 2020-05-08 21:23 n03991062_24866.cls
-r--r--r-- bigdata/bigdata 108611 2020-05-08 21:23 n03991062_24866.jpg
-r--r--r-- bigdata/bigdata      3 2020-05-08 21:23 n07749582_9506.cls
-r--r--r-- bigdata/bigdata 129044 2020-05-08 21:23 n07749582_9506.jpg
-r--r--r-- bigdata/bigdata      3 2020-05-08 21:23 n03425413_23604.cls
-r--r--r-- bigdata/bigdata 106255 2020-05-08 21:23 n03425413_23604.jpg
-r--r--r-- bigdata/bigdata      3 2020-05-08 21:23 n02795169_27274.cls
```
WebDataset datasets can be used directly from local disk, from web servers (hence the name), from cloud storage and object stores, just by changing a URL. WebDataset datasets can be used for training without unpacking, and training can even be carried out on streaming data, with no local storage.
Shuffling during training is important for many deep learning applications, and WebDataset performs shuffling both at the shard level and at the sample level. Splitting of data across multiple workers is performed at the shard level using a user-provided shard_selection function that defaults to a function that splits based on get_worker_info. (WebDataset can be combined with the tensorcom library to offload decompression/data augmentation and provide RDMA and direct-to-GPU loading; see below.)
Code Sample
Code Sample
Here are some code snippets illustrating the use of WebDataset in a typical PyTorch deep learning application (you can find a full example at http://github.com/tmbdev/pytorch-imagenet-wds.
```python
import webdataset as wds
import ...
sharedurl = "/imagenet/imagenet-train-{000000..001281}.tar"
normalize = transforms.Normalize(
  mean=[0.485, 0.456, 0.406],
  std=[0.229, 0.224, 0.225])
preproc = transforms.Compose([
  transforms.RandomResizedCrop(224),
  transforms.RandomHorizontalFlip(),
  transforms.ToTensor(),
  normalize,
])
dataset = (
  wds.Dataset(sharedurl)
  .shuffle(1000)
  .decode("pil")
  .rename(image="jpg;png", data="json")
  .map_dict(image=preproc)
  .to_tuple("image", "data")
)
loader = torch.utils.data.DataLoader(dataset, batch_size=64, num_workers=8)
for inputs, targets in loader:
  ...
  ```
for inputs, targets in loader:
  ...
  ```
This code is nearly identical to the file-based I/O pipeline found in the PyTorch Imagenet example: it creates a preprocessing/augmentation pipeline, instantiates a dataset using that pipeline and a data source location, and then constructs a DataLoader instance from the dataset.
WebDataset uses a fluent API for a configuration that internally builds up a processing pipeline. Without any added processing stages, In this example, WebDataset is used with the PyTorch DataLoader class, which replicates DataSet instances across multiple threads and performs both parallel I/O and parallel data augmentation.
WebDataset instances themselves just iterate through each training sample as a dictionary:
```python
load from a web server using a separate client process
sharedurl = "pipe:curl -s http://server/imagenet/imagenet-train-{000000..001281}.tar"
dataset = wds.Dataset(sharedurl)
for sample in dataset:
  # sample["jpg"] contains the raw image data
sample["jpg"] contains the raw image data
# sample["cls"] contains the class
  ...
  ```
For a general introduction to how we handle large scale training with WebDataset, see these YouTube videos.
Related Software

AIStore is an open-source object store capable of full-bandwidth disk-to-GPU data delivery (meaning that if you have 1000 rotational drives with 200 MB/s read speed, AIStore actually delivers an aggregate bandwidth of 200 GB/s to the GPUs). AIStore is fully compatible with WebDataset as a client, and in addition understands the WebDataset format, permitting it to perform shuffling, sorting, ETL, and some map-reduce operations directly in the storage system. AIStore can be thought of as a remix of a distributed object store, a network file system, a distributed database, and a GPU-accelerated map-reduce implementation.
tarp is a small command-line program for splitting, merging, shuffling, and processing tar archives and WebDataset datasets.


tensorcom is a library supporting distributed data augmentation and RDMA to GPU.


pytorch-imagenet-wds contains an example of how to use WebDataset with ImageNet, based on the PyTorch ImageNet example.


Bigdata 2019 Paper with Benchmarks


Check out the library and provide your feedback for RFC 38419.
layout: blog_detail
title: 'PyTorch 0.4.0 Migration Guide'
redirect_from: /2018/04/22/0_4_0-migration-guide.html

Welcome to the migration guide for PyTorch 0.4.0. In this release we introduced many exciting new features and critical bug fixes, with the goal of providing users a better and cleaner interface. In this guide, we will cover the most important changes in migrating existing code from previous versions:

Tensors and Variables have merged
Support for 0-dimensional (scalar) Tensors
Deprecation of the volatile flag
dtypes, devices, and Numpy-style Tensor creation functions
Writing device-agnostic code
New edge-case constraints on names of submodules, parameters, and buffers in nn.Module

Merging Tensor and Variable and classes
torch.Tensor and torch.autograd.Variable are now the same class. More precisely, torch.Tensor is capable of tracking history and behaves like the old Variable; Variable wrapping continues to work as before but returns an object of type torch.Tensor. This means that you don't need the Variable wrapper everywhere in your code anymore.
The type() of a Tensor has changed
Note also that the type() of a Tensor no longer reflects the data type. Use isinstance() or x.type()instead:
>>> x = torch.DoubleTensor([1, 1, 1])
>>> print(type(x))  # was torch.DoubleTensor
"<class 'torch.Tensor'>"
>>> print(x.type())  # OK: 'torch.DoubleTensor'
'torch.DoubleTensor'
>>> print(isinstance(x, torch.DoubleTensor))  # OK: True
True
True
```
When does autograd start tracking history now?
requires_grad, the central flag for autograd, is now an attribute on Tensors. The same rules previously used for Variables applies to Tensors; autograd starts tracking history when any input Tensor of an operation has requires_grad=True. For example,
```python



x = torch.ones(1)  # create a tensor with requires_grad=False (default)
x.requires_grad
False
y = torch.ones(1)  # another tensor with requires_grad=False
z = x + y
both inputs have requires_grad=False. so does the output
z.requires_grad
False
then autograd won't track this computation. let's verify!
z.backward()
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
now create a tensor with requires_grad=True
w = torch.ones(1, requires_grad=True)
w = torch.ones(1, requires_grad=True)
w.requires_grad
True
add to the previous result that has require_grad=False
total = w + z
the total sum now requires grad!
total.requires_grad
True
autograd can compute the gradients as well
total.backward()
w.grad
tensor([ 1.])
and no computation is wasted to compute gradients for x, y and z, which don't require grad
z.grad == x.grad == y.grad == None
True




#### Manipulating `requires_grad` flag

Other than directly setting the attribute, you can change this flag `in-place` using [`my_tensor.requires_grad_()`](https://pytorch.org/docs/0.4.0/tensors.html#torch.Tensor.requires_grad_), or, as in the above example, at creation time by passing it in as an argument (default is `False`), e.g.,

```python
>>> existing_tensor.requires_grad_()
>>> existing_tensor.requires_grad
True
>>> my_tensor = torch.zeros(3, 4, requires_grad=True)
>>> my_tensor.requires_grad
True

What about .data?
True
```
What about .data?
.data was the primary way to get the underlying Tensor from a Variable. After this merge, calling y = x.data still has similar semantics. So y will be a Tensor that shares the same data with x, is unrelated with the computation history of x, and has requires_grad=False.
However, .data can be unsafe in some cases. Any changes on x.data wouldn't be tracked by autograd, and the computed gradients would be incorrect if x is needed in a backward pass. A safer alternative is to use x.detach(), which also returns a Tensor that shares data with requires_grad=False, but will have its in-place changes reported by autograd if x is needed in backward.
Here is an example of the difference between .data and x.detach() (and why we recommend using detach in general).
If you use Tensor.detach(), the gradient computation is guaranteed to be correct.
```python
>>> a = torch.tensor([1,2,3.], requires_grad = True)
>>> out = a.sigmoid()
>>> c = out.detach()
>>> c.zero_()
tensor([ 0.,  0.,  0.])

>>> out  # modified by c.zero_() !!
tensor([ 0.,  0.,  0.])

>>> out.sum().backward()  # Requires the original value of out, but that was overwritten by c.zero_()
RuntimeError: one of the variables needed for gradient computation has been modified by an

However, using Tensor.data can be unsafe and can easily result in incorrect gradients when a tensor is required for gradient computation but modified in-place.
>>> a = torch.tensor([1,2,3.], requires_grad = True)
>>> out = a.sigmoid()
>>> c = out.data
>>> c.zero_()
tensor([ 0.,  0.,  0.])

>>> out  # out  was modified by c.zero_()
tensor([ 0.,  0.,  0.])

>>> out.sum().backward()
>>> a.grad  # The result is very, very wrong because `out` changed!
tensor([ 0.,  0.,  0.])

Support for 0-dimensional (scalar) Tensors
Support for 0-dimensional (scalar) Tensors
Previously, indexing into a Tensor vector (1-dimensional tensor) gave a Python number but indexing into a Variable vector gave (inconsistently!) a vector of size (1,)! Similar behavior existed with reduction functions, e.g. tensor.sum() would return a Python number, but variable.sum() would return a vector of size (1,).
Fortunately, this release introduces proper scalar (0-dimensional tensor) support in PyTorch! Scalars can be created using the new torch.tensor function (which will be explained in more detail later; for now just think of it as the PyTorch equivalent of numpy.array). Now you can do things like:
```python



torch.tensor(3.1416)         # create a scalar directly
tensor(3.1416)
torch.tensor(3.1416).size()  # scalar is 0-dimensional
torch.Size([])
torch.tensor([3]).size()     # compare to a vector of size 1
torch.Size([1])
vector = torch.arange(2, 6)  # this is a vector
vector
tensor([ 2.,  3.,  4.,  5.])
vector
tensor([ 2.,  3.,  4.,  5.])
vector.size()
torch.Size([4])
vector[3]                    # indexing into a vector gives a scalar
tensor(5.)
vector[3].item()             # .item() gives the value as a Python number
5.0
mysum = torch.tensor([2, 3]).sum()
mysum
tensor(5)
mysum.size()
torch.Size([])
```



Accumulating losses
Consider the widely used pattern total_loss += loss.data[0]. Before 0.4.0. loss was a Variable wrapping a tensor of size (1,), but in 0.4.0 loss is now a scalar and has 0 dimensions. Indexing into a scalar doesn't make sense (it gives a warning now, but will be a hard error in 0.5.0). Use loss.item() to get the Python number from a scalar.
Note that if you don't convert to a Python number when accumulating losses, you may find increased memory usage in your program. This is because the right-hand-side of the above expression used to be a Python float, while it is now a zero-dim Tensor. The total loss is thus accumulating Tensors and their gradient history, which may keep around large autograd graphs for much longer than necessary.
Deprecation of volatile flag
The volatile flag is now deprecated and has no effect. Previously, any computation that involves a Variable with volatile=True wouldn't be tracked by autograd. This has now been replaced by a set of more flexible context managers including torch.no_grad(), torch.set_grad_enabled(grad_mode), and others.
```python



x = torch.zeros(1, requires_grad=True)
with torch.no_grad():
...     y = x * 2
y.requires_grad
False
is_train = False
y.requires_grad
False
is_train = False
with torch.set_grad_enabled(is_train):
...     y = x * 2
y.requires_grad
False
torch.set_grad_enabled(True)  # this can also be used as a function
y = x * 2
y.requires_grad
True
torch.set_grad_enabled(False)
y = x * 2
y.requires_grad
False
```



dtypes, devices and NumPy-style creation functions
In previous versions of PyTorch, we used to specify data type (e.g. float vs double), device type (cpu vs cuda) and layout (dense vs sparse) together as a "tensor type". For example, torch.cuda.sparse.DoubleTensor was the Tensor type representing the double data type, living on CUDA devices, and with COO sparse tensor layout.
In this release, we introduce torch.dtype, torch.device and torch.layout classes to allow better management of these properties via NumPy-style creation functions.
torch.dtype
Below is a complete list of available torch.dtypes (data types) and their corresponding tensor types.



Data
type torch.dtype
Tensor types




32-bit floating point
torch.float32 or torch.float
torch.*.FloatTensor


64-bit floating point
torch.float64 or torch.double
torch.*.DoubleTensor


16-bit floating point
torch.float16 or torch.half
torch.*.HalfTensor
| 8-bit integer (unsigned) |  torch.uint8 | torch.*.ByteTensor
| 8-bit integer (signed) | torch.int8 | torch.*.CharTensor
| 16-bit integer (signed) | torch.int16 or torch.short | torch.*.ShortTensor
| 32-bit integer (signed) | torch.int32 or torch.int | torch.*.IntTensor
| 64-bit integer (signed) | torch.int64 or torch.long | torch.*.LongTensor
The dtype of a tensor can be access via its dtype attribute.
torch.device
A torch.device contains a device type ('cpu' or 'cuda') and optional device ordinal (id) for the device type. It can be initialized with torch.device('{device_type}') or torch.device('{device_type}:{device_ordinal}').
If the device ordinal is not present, this represents the current device for the device type; e.g., torch.device('cuda') is equivalent to torch.device('cuda:X') where X is the result of torch.cuda.current_device().
The device of a tensor can be accessed via its device attribute.
torch.layout
torch.layout represents the data layout of a Tensor. Currently torch.strided (dense tensors, the default) and torch.sparse_coo (sparse tensors with COO format) are supported.
The layout of a tensor can be access via its layout attribute.
Creating Tensors
Creating Tensors
Methods that create a Tensor now also take in dtype, device, layout, and requires_grad options to specify the desired attributes on the returned Tensor. For example,
>>> device = torch.device("cuda:1")
>>> x = torch.randn(3, 3, dtype=torch.float64, device=device)
tensor([[-0.6344,  0.8562, -1.2758],
        [ 0.8414,  1.7962,  1.0589],
        [-0.1369, -1.0462, -0.4373]], dtype=torch.float64, device='cuda:1')
>>> x.requires_grad  # default is False
False
>>> x = torch.zeros(3, requires_grad=True)
>>> x.requires_grad
True

torch.tensor(data, ...)
torch.tensor is one of the newly added tensor creation methods. It takes in array-like data of all kinds and copies the contained values into a new Tensor. As mentioned earlier, torch.tensor is the PyTorch equivalent of NumPy's numpy.arrayconstructor. Unlike the torch.*Tensor methods, you can also create zero-dimensional Tensors (aka scalars) this way (a single python number is treated as a Size in the torch.*Tensor methods). Moreover, if a dtype argument isn't given, it will infer the suitable dtype given the data. It is the recommended way to create a tensor from existing data like a Python list. For example,
```python



cuda = torch.device("cuda")
torch.tensor([[1], [2], [3]], dtype=torch.half, device=cuda)
tensor([[ 1],
        [ 2],
        [ 3]], device='cuda:0')
torch.tensor(1)               # scalar
torch.tensor(1)               # scalar
tensor(1)
torch.tensor([1, 2.3]).dtype  # type inferece
torch.float32
torch.tensor([1, 2]).dtype    # type inferece
torch.int64
```



We've also added more tensor creation methods. Some of them have torch.*_like and/or tensor.new_* variants.

torch.*_like takes in an input Tensor instead of a shape. It returns a Tensor with same attributes as the input Tensor by default unless otherwise specified:

```python



x = torch.randn(3, dtype=torch.float64)
torch.zeros_like(x)
 tensor([ 0.,  0.,  0.], dtype=torch.float64)
torch.zeros_like(x, dtype=torch.int)
 tensor([ 0,  0,  0], dtype=torch.int32)
  ```




tensor.new_* can also create Tensors with same attributes as tensor, but it always takes in a shape argument:

```python



x = torch.randn(3, dtype=torch.float64)
x.new_ones(2)
 tensor([ 1.,  1.], dtype=torch.float64)
x.new_ones(4, dtype=torch.int)
 tensor([ 1,  1,  1,  1], dtype=torch.int32)
  ```