We demonstrate that training ResNet-50 on ImageNet for 90 epochs can be achieved in 15 minutes with 1024 Tesla P100 GPUs. This was made possible by using a large minibatch size of 32k. To maintain accuracy with this large minibatch size, we employed several techniques such as RMSprop warm-up, batch normalization without moving averages, and a slow-start learning rate schedule. This paper also describes the details of the hardware and software of the system used to achieve the above performance.
Further reduction was achieved (15 minutes) using 1024 Tesla P100 GPUs in REF .
Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes
ArXiv
ArXiv
Computer Science