  The increasing size of neural network models has been critical for
improvements in their accuracy, but device memory is not growing at the same
rate. This creates fundamental challenges for training neural networks within
limited memory environments. In this work, we propose ActNN, a memory-efficient
training framework that stores randomly quantized activations for back
propagation. We prove the convergence of ActNN for general network
architectures, and we characterize the impact of quantization on the
convergence via an exact expression for the gradient variance. Using our
theory, we propose novel mixed-precision quantization strategies that exploit
the activation's heterogeneity across feature dimensions, samples, and layers.
These techniques can be readily applied to existing dynamic graph frameworks,
such as PyTorch, simply by substituting the layers. We evaluate ActNN on
mainstream computer vision models for classification, detection, and
segmentation tasks. On all these tasks, ActNN compresses the activation to 2
bits on average, with negligible accuracy loss. ActNN reduces the memory
footprint of the activation by 12x, and it enables training with a 6.6x to 14x
larger batch size.
