Convolutional neural networks [3] have proven useful in many domains, including computer vision [1, 4, 5] , audio processing [6, 7] and natural language processing [8] . These powerful models come at great cost in training time, however. Currently, long training periods make experimentation difficult and time consuming. In this work, we consider a standard architecture [1] trained on the Imagenet dataset [2] for classification and investigate methods to speed convergence by parallelizing training across multiple GPUs. In this work, we used up to 4 NVIDIA TITAN GPUs with 6GB of RAM. While our experiments are performed on a single server, our GPUs have disjoint memory spaces, and just as in the distributed setting, communication overheads are an important consideration. Unlike previous work [9, 10, 11], we do not aim to improve the underlying optimization algorithm. Instead, we isolate the impact of parallelism, while using standard supervised back-propagation and synchronous mini-batch stochastic gradient descent. We consider two basic approaches: data and model parallelism [9] . In data parallelism, the minibatch is split across several GPUs as shown in fig. 3 . Each GPU is responsible for computing gradients with respect to all model parameters, but does so using a subset of the samples in the mini-batch. This is the most straightforward parallelization method, but it requires considerable communication between GPUs, since each GPU must communicate both gradients and parameter values on every update step. Also, each GPU must use a large number of samples to effectively utilize the highly parallel device; thus, the mini-batch size effectively gets multiplied by the number of GPUs, hampering convergence. In our implementation, we find a speed-up of 1
Facebook REF adopts both data-parallelism and model-parallelism to train multi-GPU machines.
Multi-GPU Training of ConvNets
ICLR 2013
Computer Science Mathematics