Paper:
DeePa is a deep learning framework that explores parallelism in all parallelizable dimensions to accelerate the training process of convolutional neural networks. DeePa optimizes parallelism at the granularity of each individual layer in the network. We present an elimination-based algorithm that finds an optimal parallelism configuration for every layer. Our evaluation shows that DeePa achieves up to 6.5× speedup compared to state-of-the-art deep learning frameworks and reduces data transfers by up to 23×.

Review:
The paper proposes an approach that offers speedup on common convolutional neural networks. It presents the approach well and shows results comparing with other popular frameworks used in the field.
Originality
- The automation of parallelism across the different dimensions in each of the layers appears somewhat new. Although parallelism across each of the individual dimensions has been explored (batch parallel is most common and best supported, height and width is discussed at least in the DistBelief paper), automatically exploring this to find the most efficient approach is new. The splitting across channels seems not to have been covered in a paper before.
Significance
- Paper shows a significant speedup over existing approaches on a single machine (16 GPUs). It is unclear how well this would translate across machines or to more devices, and also on newer devices - the experiments were all done on 16 K80s (3 generations old GPUs). While the approach is interesting, its impact also depends on the speedup on the common hardware used today.
Pros:
- Providing better parallelism opportunities for convolutional neural networks
- Simple approach to finding optimal global configurations that seems to work well
- Positive results with significant speedups across 3 different networks
Cons:
- Unclear if speedups hold on newer devices
- Useful to see how this scales across more than 1 machine
- Claim on overlapping computation with data transfer seems incorrect. I am pretty sure TensorFlow and possibly PyTorch supports this.
Questions:
- How long does finding the optimal global configuration take for each model?