Binary and Ternary Natural Language Generation (2023)

Abstract: Ternary and binary neural networks enable multiplication-free computation and promise multiple orders of magnitude efficiency gains over full-precision networks if implemented on specialized hardware. However, since both the parameter and the output space are highly discretized, such networks have proven very difficult to optimize. The difficulties are compounded for the class of transformer text generation models due to the sensitivity of the attention operation to quantization and the noise-compounding effects of autoregressive decoding in the high-cardinality output space. We approach the problem with a mix of statistics-based quantization for the weights and elastic quantization of the activations and demonstrate the first ternary and binary transformer models on the downstream tasks of summarization and machine translation. Our ternary BART base achieves an R1 score of 41 on the CNN/DailyMail benchmark, which is merely 3.9 points behind the full model while being 16x more efficient. Our binary model, while less accurate, achieves a highly non-trivial score of 35.6. For machine translation, we achieved BLEU scores of 21.7 and 17.6 on the WMT16 En-Ro benchmark, compared with a full precision mBART model score of 26.8. We also compare our approach in the 8-bit activation setting, where our ternary and even binary weight models can match or outperform the best existing 8-bit weight models in the literature. Our code and models are available at: https://github.com/facebookresearch/Ternary_Binary_Transformer .

## introduction generative pre-trained transformers @xcite @xcite have emerged as powerful and generic tools, driving breakthroughs not only in language understanding but the field of ai in general. these models owe * equal contribution their success mainly to their seemingly infinite ability to scale to ever-larger data and model sizes. unfortunately, such scaling comes at the cost of large computational requirements, putting extensively large generative transformers out of reach of all but the most resource-rich institutions. even moderately sized pre-trained transformers have limited applications due to their size and computational cost. making generative transformers more efficient is imperative for widening their use to more devices and practical applications. in this work, we explore making generative pretrained transformers more efficient via the quantization of their weights and activations. quantizing the weights of a neural network is useful for compression and allows the model to be stored more efficiently. however, compression alone does not reduce computation costs since the network's activations need to be computed in full precision. quantizing both weights and activations allows computation to be performed with lower precision, potentially leading to significant efficiency gains depending on the quantization level and hardware implementation. quantizing neural networks have a long history, and multiple works have attempted to quantize pre-trained transformers at various quantization levels @xcite @xcite . most of this work focuses on encoder-only models (mainly bert) for sentence and token classification tasks. quantizing text generation models has generally been regarded as a more difficult task @xcite due to the large output vocabulary and sequential decoding. recent work has tackled this problem, though only for mild quantization levels (down to 8-bit activations) and with mixed success. in contrast, we are interested in very low-bit quantization, down to ternary and even binary weights and activations. in order to achieve this, we combine and unify best practices for weight and activation quantization and present a frame-work that uses gradient-matching quantization for weights and elastic quantization for activations. we apply our method to natural language generation tasks and, for the first time, demonstrate low-bit generative transformers of competitive accuracy. our ternary (weight and activation) model lags a full-precision bart @xcite model by only 4 points in rouge on the xsum summarization dataset. in contrast, our model with ternary weights and 8-bit activations comes within 1 point and even outperforms comparable state-of-the-art models with 8-bit weights. we also demonstrate a fully binary (weights and activations) model. while not as competitive, it is able to achieve a highly non-trivial rouge-1 score of 31.7. our results also extend to machine translation models. on the wmt16 en-ro benchmark, we quantize an mbart model to extend the ternaryweight 8-bit activation sota by 1.2 points while demonstrating fully ternary and fully binary translation models for the first time. we summarize our contributions as follows: • we propose a novel combination of statisticsbased weight quantization with learning-based activation quantization, which enables stably training transformer encoder-decoder models to converge in the fully ternary/binary settings, which was not previously possible. ## method in this section, we first introduce the previous practices in binarization and ternarization. then, we introduce a unified statistic-based weight binarization / ternarization method that can alleviate the gradient mismatch issue and enhance the quantized weights entropy. lastly, we analyze the difference between weight quantization and activation quantization and propose an elastic ternarization method for activations. we abbreviate our method as tbt, short for "ternary / binary transformer". ## experiments in this section, we evaluate the effectiveness of our low-bit quantization scheme for natural language generative model on text summarization benchmarks: cnn/dailymail @xcite and xsum @xcite . we additionally experiment on the machine translation task with mbart on wmt16 english-romanian (en-ro) dataset @xcite . ## conclusion we have demonstrated high accuracy ternary and binary natural language generation models based on a pre-trained transformer encoder-decoder backbone. quantizing both the weights and the activations of the network allow these models to run on special-purpose hardware using binary and ternary arithmetic, which doesn't require multiplication modules. therefore our results promise multiple orders of magnitude gains in efficiency while running these models, and can drastically expand the use cases of such models beyond just high end gpu servers. we are especially excited about the implications of our results for larger text generation models such as gpt-3 @xcite . these models have both demonstrated impressive capabilities, while also presenting enormous scaling and computational challenges. low-bit quantization is a promising approach to mitigate some of these issues. whether our approach will scale to these models is an open problem and an exciting future research direction.