  I/O is emerging as a major bottleneck for machine learning training,
especially in distributed environments such as clouds and supercomputers.
Optimal data ingestion pipelines differ between systems, and increasing
efficiency requires a delicate balance between access to local storage,
external filesystems, and remote workers; yet existing frameworks fail to
efficiently utilize such resources. We observe that, given the seed generating
the random access pattern for training with SGD, we have clairvoyance and can
exactly predict when a given sample will be accessed. We combine this with a
theoretical analysis of access patterns in training and performance modeling to
produce a novel machine learning I/O middleware, HDMLP, to tackle the I/O
bottleneck. HDMLP provides an easy-to-use, flexible, and scalable solution that
delivers better performance than state-of-the-art approaches while requiring
very few changes to existing codebases and supporting a broad range of
environments.
