  We introduce CryptGPU, a system for privacy-preserving machine learning that
implements all operations on the GPU (graphics processing unit). Just as GPUs
played a pivotal role in the success of modern deep learning, they are also
essential for realizing scalable privacy-preserving deep learning. In this
work, we start by introducing a new interface to losslessly embed cryptographic
operations over secret-shared values (in a discrete domain) into floating-point
operations that can be processed by highly-optimized CUDA kernels for linear
algebra. We then identify a sequence of "GPU-friendly" cryptographic protocols
to enable privacy-preserving evaluation of both linear and non-linear
operations on the GPU. Our microbenchmarks indicate that our private GPU-based
convolution protocol is over 150x faster than the analogous CPU-based protocol;
for non-linear operations like the ReLU activation function, our GPU-based
protocol is around 10x faster than its CPU analog.
  With CryptGPU, we support private inference and private training on
convolutional neural networks with over 60 million parameters as well as handle
large datasets like ImageNet. Compared to the previous state-of-the-art, when
considering large models and datasets, our protocols achieve a 2x to 8x
improvement in private inference and a 6x to 36x improvement for private
training. Our work not only showcases the viability of performing secure
multiparty computation (MPC) entirely on the GPU to enable fast
privacy-preserving machine learning, but also highlights the importance of
designing new MPC primitives that can take full advantage of the GPU's
computing capabilities.
