Assigning Distinct Roles to Quantized and Low-Rank Matrices Toward Optimal Weight Decomposition (2025)

Abstract: Decomposing weight matrices into quantization and low-rank components ( \bf W\approx Q+LR ) is a widely used technique for compressing large language models (LLMs). Existing joint optimization methods iteratively alternate between quantization and low-rank approximation. However, these methods tend to prioritize one component at the expense of the other, resulting in suboptimal decompositions that fail to leverage each component’s unique strengths. In this work, we introduce Outlier-Driven Low-Rank Initialization (ODLRI), which assigns low-rank components the specific role of capturing activation-sensitive weights. This structured decomposition mitigates outliers’ negative impact on quantization, enabling more effective balance between quantization and low-rank approximation. Experiments on Llama2 (7B, 13B, 70B), Llama3-8B, and Mistral-7B demonstrate that incorporating ODLRI into the joint optimization framework consistently reduces activation-aware error, minimizes quantization scale, and improves perplexity and zero-shot accuracy in low-bit settings.

## introduction quantization @xcite @xcite and weight matrix factorization @xcite are two widely used techniques for compressing large language models (llms) to enable efficient inference on resource-constrained hardware. post-training quantization (ptq) reduces model size and computational cost by mapping high-precision weights to lower-bit representations @xcite @xcite , while matrix factorization approximates weight matrices with compact factored representations @xcite . recent joint optimization approaches combine these methods to achieve extreme compression, representing † correspondence to: albert @xmath0.ac.@xmath1: w ≈ q + lr @xcite @xcite . joint optimization approaches minimize representation error by iteratively alternating between quantization and low-rank approximation. these methods typically adopt either a quantize-first strategy @xcite or a lowrank-first strategy @xcite . while these differ in iteration ordering, they can be equivalently understood as distinct initialization choices for the low-rank components: the quantize-first approach initializes lr to zero, while the low-rank-first approach initializes them using factorized weights. critically, existing methods treat these different orderings merely as preparatory steps, assuming iterative updates will naturally converge to optimal solutions. however, viewing this process through the lens of initialization reveals an underexplored aspect of how initialization strategies might fundamentally affect weight decomposition quality. in this work, we investigate the role of initialization in joint optimization. our analysis shows that different initializations lead to distinct solution spaces, with components maintaining persistent roles throughout optimization. quantize-first methods treat low-rank components as error correction terms, while low-rank-first methods preserve them as the primary weight representation. this finding highlights that initialization fundamentally determines the role assignment between quantization and low-rank components, raising a key question: what is the optimal initialization strategy for decomposing weights into quantized and low-rank matrices? recent works have shown that quantization errors are pronounced for weights associated with activation outliers, as extreme activations amplify weight sensitivity @xcite @xcite @xcite . building on this insight, we introduce outlier-driven low-rank initialization (odlri), which assigns a specific role to the low-rank component to capture these salient weights while using the quantized matrix to express the residuals. by handling outlier-sensitive weights through the low-rank component, our approach stabilizes quantization and enables more precise weight decomposition. through extensive experiments on llama2 (7b, 13b, 70b) @xcite , llama3-8b @xcite , and mistral-7b @xcite , we demonstrate that incorporating odlri into the joint optimization framework consistently improves perplexity and zero-shot accuracy across extreme low-bit settings. our analysis shows that odlri reduces activation-aware error, minimizes quantization scale, and enhances model performance, highlighting the importance of structured initialization in low-bit quantization. these results present a principled approach for stable and efficient llm compression, and provide new insights into the effective decomposition of quantization and low-rank matrices. we summarize our contributions as follows: ## experiment 4.1 experimental setup to evaluate the effectiveness of outlier-driven low-rank initialization (odlri), we integrate it into caldera @xcite , an iterative joint optimization framework for extreme low-bit quantization of q, l, r. by default, caldera first quantizes the weight matrix and then optimizes a low-rank component, effectively treating l, r as zero-initialized correction factors. in contrast, odlri replaces this zeroinitialization with an outlier-aware initialization, ensuring that salient weights are explicitly modeled in the low-rank component before quantization. we conduct a comprehensive comparison against caldera's standard setup to quantify the impact of lr initialization on final accuracy and stability. quantization setup. we integrate odlri into caldera while keeping the quantization configuration largely unchanged. for quantize, we use a 2-bit quantization scheme implemented via quip# @xcite , which employs an e8 lattice codebook for s caldera performs outer iterations alternating between quantize and lrapprox, along with inner iterations for the lplr procedure. we follow caldera's default configuration, running 15 outer iterations, and 10 inner iterations when the l, r components are quantized to 4-bit. ## conclusion in this work, we investigated the role of initialization strategies in iterative joint optimization approaches for llm compression. we introduce a unified framework that reformulates joint optimization algorithms through the lens of low-rank component initialization. our analysis shows that the choice of initialization determines the entire trajectory of weight decomposition by assigning persistent roles to the components. based on these insights, we proposed odlri, which assigns a distinct role to the low-rank component to capture activation-sensitive weights while using quantization for the remaining weights. through extensive experiments across various llm architectures, we demonstrated that incorporating odlri significantly improves model performance and compression stability. consequently, our approach advances the practical implementation of efficient llm compression and steers us toward optimal weight decomposition. acknowledgement this work was supported by institute of information & communications technology planning & evaluation (iitp) grant funded by the korea government(msit) (no. rs-2024-00457882, national ai research lab project), iitp grant funded by the korean government (msit) (no. rs-2020-ii201361, artificial intelligence graduate school program (yonsei university)), and k-chips(korea collaborative & high-tech initiative for prospective semiconductor research) (rs-2024-00405946, 24052-15tc) funded by the ministry of trade, industry & energy (motie, korea).