The enormous amount of data and computation required to train DNNshave led to the rise of various parallelization strategies . Broadly, there are two strategies: 1) Data-Parallelism -- replicating the DNN on multiple processes and training on different training samples, and 2) Model-Parallelism -- dividing elements of the DNN itself into partitions across different processes . While data-parallelism has been extensively studied and developed,  model-parallelism has received less attention as it is non-trivial to split the model across processes . In this paper, we propose HyPar-Flow : a framework for scalable and user-transparent parallel training of very large DNNs (up to 5, 000 layers). We exploit TensorFlow's Eager Execution features and KerasAPIs for model definition and distribution . HyPar-Flow exposes a simple API to offer data, model, and hybrid (model + data) parallel training for models defined using the Keras API. Under the hood, we introduce MPI communication primitives like send and recv on layer boundaries for data exchange between model-partitions and allreduce for gradient exchange across model-replicas. Our proposed designs in HyPar-Flow offer up to 3.1x speedup over sequential training for ResNet-110 and up to 1.6x speedup over Horovod-based data-parallel training for ResNet-1001; a model that has 1, 001 layers and 30 million parameters. We provide an in-depth performance characterization of the HyPar-Flow framework on multiple HPC systems with diverse CPU architectures including Intel Xeon(s)and AMD EPYC. HyPar-Flow provides 110x speed up on 128 nodes of the Stampede2 cluster at TACC for hybrid-parallel training of ResNet-1001 .
To reduce training time of large-scale DNNs, scientists have started to explore parallelization strategies like data-parallelism, model-parallelism, and hybrid-parallelism . While data-parallelism has been extensively studied and developed, several problems exist in realizing model-parallelism and hybrid-parallelism efficiently. Four major problems we focus on are: 1) defining a notion of a distributed model across processes , 2) implementing forward/back-propagation across process boundaries that requires explicit communication, 3) obtaining parallel speedup on an inherently sequential task, and 4) achieving scalability without losing out on a model's accuracy. To address these problems, we create HyPar-Flow --- a model-size/-type agnostic, scalable, practical, and user-transparent system for hybrid-parallel training by exploiting MPI, Keras, and TensorFlow . HyPar-Flow provides a single API that can be used to perform data, model, and hybrid parallel training of any Keras model at scale. We create an internal distributed representation of the user-provided Keras model, utilize TF's Eager execution features for distributed forward/back-propagation across processes, exploit pipelining to improve performance and leverage efficient MPI primitives for scalable communication. Between model partitions, we use send and recv to exchange layer-data/partial-errors while allreduce is used to accumulate/average gradients across model replicas. Beyond the design and implementation of HyPar-Flow, we also provide comprehensive correctness and performance results on three state-of-the-art HPC systems including TACC Frontera (#5 on URL). For ResNet-1001, an ultra-deep model, HyPar-Flow provides: 1) Up to 1.6x speedup over Horovod-based data-parallel training, 2) 110x speedup over single-node on 128 Stampede2 nodes, and 3) 481x speedup over single-node on 512 Frontera nodes .
R
The enormous amount of data and computation required to train DNNshave led to the rise of various parallelization strategies . Broadly, there are two strategies: 1) Data-Parallelism -- replicating the DNN on multiple processes and training on different training samples, and 2) Model-Parallelism -- dividing elements of the DNN itself into partitions across different processes
To reduce training time of large-scale DNNs, scientists have started to explore parallelization strategies like data-parallelism, model-parallelism, and hybrid-parallelism
A
several problems exist in realizing
R
has received less attention as it is non-trivial to split the
and hybrid-parallelism efficiently. Four major problems we focus on are: 1) defining a notion of a distributed
R
. In this paper, we propose
, 2) implementing forward/back-propagation across process boundaries that requires explicit communication, 3) obtaining parallel speedup on an inherently sequential task, and 4) achieving scalability without losing out on a model's accuracy. To address these problems, we create
R
: a framework for scalable
--- a model-size/-type agnostic, scalable, practical,
R
parallel training of very large DNNs (up to 5, 000 layers). We exploit TensorFlow's Eager Execution features and KerasAPIs for model definition and distribution
system for hybrid-parallel training by exploiting MPI, Keras, and TensorFlow
R
exposes a simple API to offer
provides a single API that can be used to perform
R
(model + data) parallel training for models defined using the Keras API. Under the hood, we introduce MPI communication primitives like
parallel training of any Keras model at scale. We create an internal distributed representation of the user-provided Keras model, utilize TF's Eager execution features for distributed forward/back-propagation across processes, exploit pipelining to improve performance and leverage efficient MPI primitives for scalable communication. Between model partitions, we use
R
on layer boundaries for data exchange between model-partitions and allreduce for gradient exchange across model-replicas. Our proposed designs in
to exchange layer-data/partial-errors while allreduce is used to accumulate/average gradients across model replicas. Beyond the design and implementation of HyPar-Flow, we also provide comprehensive correctness and performance results on three state-of-the-art HPC systems including TACC Frontera (#5 on URL). For ResNet-1001, an ultra-deep model,
R
offer up to 3.1x speedup over sequential training for ResNet-110 and up to
provides: 1) Up to
R
training for ResNet-1001; a model that has 1, 001 layers and 30 million parameters. We provide an in-depth performance characterization of the HyPar-Flow framework on multiple HPC systems with diverse CPU architectures including Intel Xeon(s)and AMD EPYC. HyPar-Flow provides
training, 2)
R
speed up
speedup over single-node
D
nodes of the
R
cluster at TACC for hybrid-parallel training of ResNet-1001
nodes, and 3) 481x speedup over single-node on 512 Frontera nodes