High Performance IO For Large Scale Deep Learning. Training deep learning (DL) models on petascale datasets is essential for achieving competitive and stateoftheart performance in applications such as speech, video analytics, and object recognition. However, existing distributed filesystems were not developed for the access patterns and usability requirements of DL jobs. In this paper, we describe AIStore, a highly scalable, easytodeploy storage system, and WebDataset, a standardsbased storage format and library that permits efficient access to very large datasets. We compare system performance experimentally using image classification workloads and storing training data on a variety of backends, including local SSDs, singlenode NFS, and two identical baremetal clusters: HDFS and AIStore.

Accelerating Data Loading in Deep Neural Network Training Data loading can dominate deep neural network training time on largescale systems. We present a comprehensive study on accelerating data loading performance in largescale distributed training. We first identify performance and scalability issues in current data loading implementations. We then propose optimizations that utilize CPU resources to the data loader design. We use an analytical model to characterize the impact of data loading on the overall training time and establish the performance trend as we scale up distributed training. Our model suggests that IO rate limits the scalability of distributed training, which inspires us to design a localityaware data loading method. By utilizing software caches, our method can drastically reduce the data loading communication volume in comparison with the original data loading implementation. Finally, we evaluate the proposed optimizations with various experiments. We achieved more than 30x speedup in data loading using 256 nodes with 1,024 learners.