How Does Quantization Affect Multilingual LLM s? (2024)

Abstract: Quantization techniques are widely used to improve inference speed and deployment of large language models. While a wide body of work examines the impact of quantization on LLMs in English, none have evaluated across languages. We conduct a thorough analysis of quantized multilingual LLMs, focusing on performance across languages and at varying scales. We use automatic benchmarks, LLM-as-a-Judge, and human evaluation, finding that (1) harmful effects of quantization are apparent in human evaluation, which automatic metrics severely underestimate: a 1.7% average drop in Japanese across automatic tasks corresponds to a 16.0% drop reported by human evaluators on realistic prompts; (2) languages are disparately affected by quantization, with non-Latin script languages impacted worst; and (3) challenging tasks like mathematical reasoning degrade fastest. As the ability to serve low-compute models is critical for wide global adoption of NLP technologies, our results urge consideration of multilingual performance as a key evaluation criterion for efficient models.

## introduction multilingual large language models (llms) have the power to bring modern language technology to the world, but only if they are cheap and reliable. known as the low-resource double bind, underserved languages and severe compute constraints often geographically co-occur @xcite , meaning that for wide adoption, multilingual llms must be highly-performant and lightweight. with the shift towards large models, quantization is a widely adopted technique to reduce cost, improve inference speed, and enable wider deployment of llms. work on quantization, however, is by-and-large evaluated in english only (e.g. @xcite @xcite . no works to our knowledge have characterized the impact of quantization on the multilingual generation capabilities expected from modern llms. ubiquitous use of compression techniques in the real world drives urgency to the question how are multilingual models impacted? our question is timely, given recent work showing that compression techniques such as quantization and sparsity amplify disparate treatment of long-tail features, which may have implications for under-represented languages in multilingual llms @xcite @xcite . indeed, many model designs choices implicitly overfit to a handful of resource rich languages: from tokenizer choice, to weighting of training data, and to widely-used quantization techniques. focusing on a small subset of high-resource languages in design degrades model performance for overlooked languages @xcite @xcite @xcite , introduces security vulnerabilities @xcite @xcite @xcite , and unfairly passes high costs to non-english users faced with high latency @xcite @xcite @xcite . we analyze four state-of-the-art (sota) multilingual llms across 3 different sizes ranging from 8 to 103 billion parameters and covering up to 23 languages, under various quantization techniques. critically, it is vital that we move beyond automatic evaluation and gather real human feedback on performance cost. we thus perform multilingual human evaluation on challenging real-world prompts in addition to llm-as-a-judge and evaluation on standard automatic benchmarks such as multilingual mmlu @xcite , mgsm @xcite . across experimental set-ups we find that: degradation on automatic metrics appears negatively correlated with training data set size, and non-latin script languages are more harmed on average. across tasks, latin-script languages scored -0.7% relative to fp16 for a 103b parameter model while non-latin scripts scored -1.9%. for a smaller 8-billion parameter model, scores were -3.0% vs. -3.7%. 3. challenging tasks degrade fastest. mathematical reasoning (-13.1%), performance on real-world challenging prompts judged by humans (-10.5%), and llm-as-a-judge (-25.9%) results are severely degraded. 4. occasionally, quantization brings benefits. similar to @xcite 's finding on english tasks, we find that quantization benefits multilingual model performance in some cases: e.g., an average 1.3% boost across tasks for a 35b model quantized with w8a8. this aligns with findings on the benefit of other compression methods such as sparsity @xcite . as the first to broadly study the impact of quantization on multilingual llms, our work is part of a wider body of literature that considers the impact of model design choices on downstream performance. our results urge attention to multilingual performance at all stages of system design. ## background quantization compresses the weights and potentially activations of a neural network to lower-bit representations. compression can be done by training the model at lower precision, known as quantization aware training (qat), or performed on the final model weights, known as post training quantization (ptq). given the difficulties in training llms especially at precision lower than 16-bits floating point, ptq methods which perform the quantization single-shot without needing gradient updates are highly desirable. training is completed at higher precision, then weights/activations are quantized without further training. in this work, we focus on post-training quantization because of its simplicity and applicability at scale. ptq of llms can be further categorized into: weight-only quantization weight matrices are quantized offline and the compressed matrices are loaded from memory during inference. quantized weight matrices have a smaller memory footprint compared to fp16 (2× smaller for 8-bit and almost 4× smaller for 4-bit), enabling inference with less compute. in memory-bound scenarios, it also enables faster inference due to fewer bytes transferred from gpu memory to the compute units. for a weight matrix @xmath0, if only a single scaling factor is used for naive quantization (per-tensor), then the quantized weights are given by: a single scaling factor might not be enough if the distribution of parameters in the weight matrix has high variance; thus one could increase the granularity of quantization by using a scale for each output dimension (per-column), i.e., ∆ ∈ r dout . however, when n is aggressively lowered to 4 bits or lower, even per-column granularity might be insufficient to cover the range of values in a column. the granularity can be further increased by using a shared scale for a subset of input dimensions called groups (g), thus the scale ∆ ∈ r d in g ×dout . a commonly used group size is 128. equation 1 gives the simplest way to quantize the weights. for @xmath1, using more advanced weight-only quantization methods like gptq @xcite or awq @xcite leads to better downstream performance. weight-and-activation quantization as the name suggests, weight-and-activation quantization quantizes the model activations alongside the weights. unlike weight-only quantization where weights can be quantized offline, quantization of activations happens at runtime. one could compute the quantization scales for various activations by using a small slice of training or validation data (static scaling) but this method typically has large degradation @xcite . for minimal degradation, it is preferred to calculate the quantization scaling factor dynamically (dynamic scaling) for each input on-the-fly. while quantizing activations is more difficult, reducing the precision of the activations alongside the weights enables the usage of specialized low-precision matrix multiplication hardware in modern gpus leading to up to 2× improvement in throughput. for a weight matrix @xmath2, naive weight-and-activation quantization with pertoken input granularity and per-column weight granularity generates output @xmath3: quantization for command r/r+ (35b/103b), we evaluate weight-only quantization at 8-bit (w8 with per-column scaling) and 4-bit (w4-g with group-wise scaling using gptq @xcite ), as well as weight-and-activation quantization at 8-bit (w8a8 with per-column scaling for weights and per-token scaling for activations). when trained with the right hyper-parameters, naive weight-and-activation quantization has minimal degradation @xcite . otherwise, smoothquant @xcite may smoothen the activation distributions to be more amenable to quantization. we explore w8a8-smoothquant (w8a8 with smoothquant) for command r+ (103b) and a 4-bit weight-only quantized variant with column-wise scaling (w4) to understand the impact of scaling granularity at extremely low-bit precision. we use 128 english samples for calibration for smoothquant and gptq @xcite . for aya 23 8b and 35b, we use bitsandbytes foot_2 for 8-bit and 4-bit quantization. this uses llm.int8() @xcite -similar to w8a8 except it performs some computations in fp16. the 4-bit uses the nf4 datatype @xcite to perform quantile quantization which limits degradation at the expense of inference speedups. ## results to clearly see the many-faceted impact of quantization, we discuss our results by quantization level ( §4.1), by task ( §4.2), by language ( §4.3), by model size ( §4.4), and by quantization strategy ( §4.5). we 10 turbo (gpt-4-1106-preview): https://platform. openai.com/docs/models/gpt-4-turbo-and-gpt-4 then report llm-as-a-judge and rm-as-a-judge ( §4.6) and human evaluation results ( §4.7). ## conclusion & future work we examine widely adopted quantization techniques for model compression and ask, how does quantization impact different languages? we perform an extensive study in state-of-the-art multilingual llms-from 8 billion to 103 billion parameters-in 20+ languages using automatic metrics, llm/rm-as-a-judge, and human evaluation. we find that: (1) damage from quantization is much worse than appears from automatic metrics: even when not observed automatically, human evaluators notice it. (2) quantization affects languages to varying degrees, with non-latin script languages more severely affected on automatic benchmarks. our results urge attention to multilingual performance at all stages of system design and might be extended to consider, for instance, languages excluded from training and out-of-distribution tasks. by minding the impact on long-tail features, we'll build better systems to serve the world.