  Deep learning researchers and practitioners usually leverage GPUs to help
train their deep neural networks (DNNs) faster. However, choosing which GPU to
use is challenging both because (i) there are many options, and (ii) users
grapple with competing concerns: maximizing compute performance while
minimizing costs. In this work, we present a new practical technique to help
users make informed and cost-efficient GPU selections: make performance
predictions using the help of a GPU that the user already has. Our technique
exploits the observation that, because DNN training consists of repetitive
compute steps, predicting the execution time of a single iteration is usually
enough to characterize the performance of an entire training process. We make
predictions by scaling the execution time of each operation in a training
iteration from one GPU to another using either (i) wave scaling, a technique
based on a GPU's execution model, or (ii) pre-trained multilayer perceptrons.
We implement our technique into a Python library called Surfer and find that it
makes accurate iteration execution time predictions on ResNet-50, Inception v3,
the Transformer, GNMT, and DCGAN across six different GPU architectures. Surfer
currently supports PyTorch, is easy to use, and requires only a few lines of
code.
