LLM - QAT : Data-Free Quantization Aware Training for Large Language Models (2024)

Abstract: Several post-training quantization methods have been applied to large language models (LLMs), and have been shown to perform well down to 8-bits. We find that these methods break down at lower bit precision, and investigate quantization-aware training for LLMs (LLM-QAT) to push quantization levels even further. We propose a data-free distillation method that leverages generations produced by the pre-trained model, which better preserves the original output distribution and allows quantizing any generative model independent of its training data, similar to post-training quantization methods. In addition to quantizing weights and activations, we also quantize the KV cache, which is critical for increasing throughput and supporting long sequence dependencies at current model sizes. We experiment with LLaMA models of sizes 7B, 13B, and 30B, at quantization levels down to 4-bits. We observe large improvements over training-free methods, especially in the low-bit settings.

## introduction following gpt-3 @xcite , several families of large language models (llms) such as opt @xcite , palm @xcite , bloom @xcite , chinchilla @xcite and llama @xcite have established that increasing model size leads to improved model capabilities. as a result, language models with tens of billions or even hundreds of billions of parameters have become the norm in today's ai landscape. despite the growing excitement around llms, serving such models to the benefit of billions of users faces significant hurdles due to their large computational cost and environmental footprint. fortunately, there has been an increasing effort to accurately quantize llms, with multiple recent figure 1 : employing llm-qat, accuracy improves by 3.0 and 1.4 points for llama-7b and llama-30b models over sota ptq under w4a8kv8 settings. additionally, w4a8kv16 llama-13b outperforms w8a8kv16 llama-7b by 1.1 points with similar ops. these improvements are substantial, especially when considering that the llama-13b model surpasses the performance of the 7b model by a mere 1.8 points. the quantization settings are w4a8kv8 / w4a8kv16 / w8a8kv8 / w8a8kv16 from left to right. see to our knowledge, qat for llms has not been investigated before. this is understandable for two reasons. first, llm training is technically difficult and resource-intensive. second, qat needs training data, which for llms is difficult to obtain. the sheer scale and diversity of pre-training data is itself an obstacle. pre-processing might be prohibitive, or worse, some data might simply not be available due to legal restrictions. it is also increasingly common to train llms in multiple stages, involving instruction tuning and reinforcement learning @xcite , which would be very difficult to replicate during qat. in this work, we side-step this issue by using generated data from the llm itself for knowledge distillation. this simple workaround, which we refer to as data-free knowledge-distillation is applicable to any generative model independent of whether or not the original training data is available. we show that this method is better able to preserve the original model's output distribution, even compared to training on large subsets of the original training set. moreover, we can successfully distill quantized models using only a small set (100k) of sampled data, thus keeping computational costs reasonable. all of our experiments are conducted using a single 8-gpu training node. as a result, we are able to distill the 7b, 13b and 30b llama models with activations quantized down to 8 bits, weights, and kv cache down to 4-bits. in this regard, our approach exhibits significant enhancements in quality compared to post-training quantization. notably, larger models employing qat outperform smaller models utilizing floating-point 16-bit representations, despite having similar model sizes, as illustrated in figure 1 . furthermore, we have successfully quantized activations to 6-bit precision, surpassing what was possible with existing methods. for a comprehensive analysis of our experimental results and detailed ablations, please refer to section 3. in summary, we present the first application of qat to llms, resulting in the first accurate 4-bit quantized llms. we also demonstrate quantizing the kv cache simultaneously with weights and activations, which is critical to alleviate throughput bottlenecks for long sequence generation. all of this is achieved by a novel data-free distillation method, which makes qat practical for large pretrained generative models. ## method quantizing large language models (llms) using quantization-aware training (qat) is a nontrivial task with challenges in two key aspects. first, llms are pre-trained to excel in zero-shot generalization, and it is crucial to preserve this capability after quantization. therefore, selecting an appropriate fine-tuning dataset is important. if the qat data is too narrow in a domain or significantly different than the original pre-training distribution, this is likely to hurt model performance. on the other hand, it is difficult to replicate the original training setup exactly, due to the scale and complexity of llm training. in section 2.1, we introduce data-free quantization-aware training (qat) which produces qat data using next token data generation. this method demonstrates superior performance compared to using subsets of the original pre-training data. second, llms exhibit unique weight and activation distributions characterized by a significant presence of outliers, which distinguishes them from smaller models. consequently, the state-of-the-art quantization clipping methods for small models do not work out of the box for llms. in section 2.2, we identify suitable quantizers for llms. ## experiments we assess the effectiveness of our approach by conducting experiments on llama-7b/13b/30b mod-els and presenting results on various tasks. specifically, we report the zero-shot performance on common sense reasoning tasks such as boolq @xcite , piqa @xcite , siqa @xcite , hellaswag @xcite , winogrande @xcite , arc @xcite , and obqa @xcite . we also assess the few-shot performance on trivi-aqa @xcite and mmlu @xcite datasets, along with perplexity scores on wikitext2 @xcite and c4 @xcite datasets. ## conclusion and limitations we proposed data-free quantization-aware training for llms and showed accurate, 4-bit quantization is possible using this technique. given the generality of the training-data-agnostic distillation method, and the growing cost of llm deployments, we expect our method to have wide applicability. for instance, the method could also be used for models trained in several stages, e.g. with instruction tuning or reinforcement learning @xcite . we leave this investigation to future work. siqa @xcite aims to measure the social and emotional intelligence of computational models through multiple choice question answering (qa). hellaswag @xcite is a benchmark for physically situated commonsense natural language inference. it consists the four-way multiple-choice problems that are trivial for humans (> 95% accuracy), but challenging for the language models. winogrande @xcite is a benchmark for commonsense reasoning. it comprises a set of 273 expert-crafted pronoun resolution problems originally designed to be unsolvable for statistical models that rely on selectional preferences or word associations. arc @xcite , the ai2 reasoning challenge, contains a collection of 7787 natural science questions. it is partitioned into a challenge set and an easy set, where the challenge set contains only questions answered incorrectly by both a retrieval-based algorithm and a word co-occurrence algorithm. obqa @xcite is a dataset of about 6000 questions for open book question answering. the task focuses on the challenge of combining a corpus of provided science facts (open book) with external broad common knowledge. a.4.2 few-shot tasks triviaqa @xcite is a closed-book question answering benchmark. it contains over 650k question-answer evidence triples, that are derived by combining 95k trivia enthusiast authored questionanswer pairs with on average six supporting evidence documents per question. mmlu @xcite , the massive multitask language understanding(mmlu) benchmark @xcite , consists of multiple choice questions covering various domains of knowledge, including humanities, stem and social sciences. a.5 generation tasks wikitext2 @xcite is a collection of over 100 million tokens extracted from the set of verified good and featured articles on wikipedia. c4 @xcite , abbreviate for colossal clean crawled corpus. common crawl foot_1 is a publiclyavailable web archive that provides "web extracted text" by removing markup and other non-text content from the scraped html files. c4 filters common crawl's web-extracted text and produces a collection of text that comprises clean and natural english text. a.6 generated data from llama-7b in this section, we show some examples of the generated data obtained through the next token generation with llama-7b model. in the mid-20th century, there was growing awareness that in a large number of industrial countries the population was aging. this awareness, along with advances in social security in many countries and a sense of urgency to avoid future generations of poor, has led to social policy being implemented that targets the elderly. the elderly often have a special status and enjoy particular social benefits. such benefits include a higher age entitlement threshold, higher pensions, early-retirement benefits, employment quotas, and free or subsidized health care. in addition, in some countries the elderly are exempt from federal and, in some cases, from state taxes. \n since most governments have set limits on their ability to pay out more in social security and health care benefits to any one group, such benefits are generally reserved for the retired and the disabled. accordingly, elderly care for the mentally disabled, infirm, and cognitively impaired has taken on particular importance, as has care for the homeless. in addition, those who are economically active can also benefit from programs, although there is the perception that the economically active are less in need of care assistance, because their families often take care of them. elderly care is generally provided on an unpaid basis, although those in need of care may have to sell their homes because of the high cost of care.\nsee also: economy; old age and retirement welfare systems; unemployment; and unemployment and employment security systems.\nbell, ian r. s., the growing crisis of old age. london: pall mall press, 1913. when my brother returned from abroad, he went with his wife to the motherland. they had many elder brothers around there, so that we all went with them to my parents\u2019 home. i, being the youngest, went first. my brother, seeing the dowry that i brought with me, laughed. he was very proud. \u2018this girl is from my sister; she is the same as my sister. why should i have her?\u2019 he said to them. but at that time my father was ill; he lay stretched on a bed before us. \u2018this is my sister; she is my sister. i did not give her to you; why should i allow you to kill my sister?\u2019 the father said to my brother. his mother said, \u2018i shall put this girl\u2019s hand in your hand. when you wish to give it away, it will be like your sister\u2019s.\u2019 my brother thought, \u2018there surely cannot be two sisters.\u2019 i told him, \u2018the old man will die soon. i have brought him his dowry of ten yens of silver. it is no small gift for him. if i do you the favour now, will you not have respect for the father when he is dead and gone? a town, and the largest inland city of the ancient aztlan (aztatlan, azatlan) located near the modern ruin of santa anas, sonora, mexico.\natsa was a settlement among the 13 tamoanchan cities of the nahua people that established their settlement in northern mexico by about 1000 a.d. the city of atsa (its people atsatla) belonged to ẗhe twelve tribes of tamoanchan ¨, which may be read in the aztec codices as tlascallaänd which are the tlascalanpeoples of the 13 cities or a confederation of peoples that were among the dominant rulers of the valley of mexico, and of western mexico in general, during the pre-eminent period of aztec civilization.\natsa as well as the other tamoancan cities of the aztecs, is said to be on a plain of jade and silver and to have a number of large, flat-roofed buildings. these buildings are described as being in ruins in 1519, during hernan cortes's first visit to mexico. although it is not known if the tamoancan cities were named before or after contact with the spaniards, it may be that the descriptions are only describing aztec ideas of the 13 cities by its inhabitants. atsa is said to be located at a point where a landform resembling a large lake (a mountain ), was formed at some point in their past that has since dried. it is here that it is said the atsatla people established their community. this article appeared in the ceylon today on 24th january 2016.\ni have been an anglican priest and a member of various committees in the ceylon evangelical lutheran church (celc) for many years. i have observed that most of the celc clergy and laity are engaged in one or the other form of mission; whether as pastor, deacon, sunday school teacher, a member of the christian education council (cec), church council or its equivalent. but do we realise the full significance of the words of the apostle paul in the first reading?\nthe readings for this sunday continue in paul\u2019s second letter to the church in corinth, where he warns the christians against the excesses of the eucharist. he warns them not to drunkenness during the eucharist. he also writes: \u201cdo not, for the sake of food, destroy the work of god\u201d (6:12).\nthese were the words paul used in his letter to his beloved church: \u201ceat anything god has created to make you healthy.\nyou should not feel guilty about eating any of these foods, but you should not eat them if they cause other people to stumble\u201d (romans 14:14-23, ceb).\nif we want to do god\u2019s will, we will learn what pleases him. (eph. 5:10). if we want others to respect god\u2019s law, then we must avoid even the appearance of sin (1 thess. 5:22). in other words, we must seek to please god in all of our actions.\nas catholics we celebrate the holy mass by taking communion of the bread and wine, in the name of the father, the son and the holy spirit to remind us of our call to love and serve one another.\nst. paul said: \u201ca little yeast spreads through the whole batch of dough\u201d (gal. 5:9). christians should celebrate god\u2019s love in a special way. we are called to live for the next generation. if we drink to excess at the eucharist it ruins the lives of our children.