  Recently, Deep Neural Networks (DNNs) have recorded great success in handling
medical and other complex classification tasks. However, as the sizes of a DNN
model and the available dataset increase, the training process becomes more
complex and computationally intensive, which usually takes a longer time to
complete. In this work, we have proposed a generic full end-to-end hybrid
parallelization approach combining both model and data parallelism for
efficiently distributed and scalable training of DNN models. We have also
proposed a Genetic Algorithm based heuristic resources allocation mechanism
(GABRA) for optimal distribution of partitions on the available GPUs for
computing performance optimization. We have applied our proposed approach to a
real use case based on 3D Residual Attention Deep Neural Network (3D-ResAttNet)
for efficient Alzheimer Disease (AD) diagnosis on multiple GPUs. The
experimental evaluation shows that the proposed approach is efficient and
scalable, which achieves almost linear speedup with little or no differences in
accuracy performance when compared with the existing non-parallel DNN models.
