  Low-bit quantization of network weights and activations can drastically
reduce the memory footprint, complexity, energy consumption and latency of Deep
Neural Networks (DNNs). However, low-bit quantization can also cause a
considerable drop in accuracy, in particular when we apply it to complex
learning tasks or lightweight DNN architectures. In this paper, we propose a
training procedure that relaxes the low-bit quantization. We call this
procedure \textit{DNN Quantization with Attention} (DQA). The relaxation is
achieved by using a learnable linear combination of high, medium and low-bit
quantizations. Our learning procedure converges step by step to a low-bit
quantization using an attention mechanism with temperature scheduling. In
experiments, our approach outperforms other low-bit quantization techniques on
various object recognition benchmarks such as CIFAR10, CIFAR100 and ImageNet
ILSVRC 2012, achieves almost the same accuracy as a full precision DNN, and
considerably reduces the accuracy drop when quantizing lightweight DNN
architectures.
