CARVQ : Corrective Adaptor with Group Residual Vector Quantization for LLM Embedding Compression (2025)

Abstract: Large Language Models (LLMs) typically rely on a large number of parameters for token embedding, leading to substantial storage requirements and memory footprints. In particular, LLMs deployed on edge devices are memory-bound, and reducing the memory footprint by compressing the embedding layer not only frees up the memory bandwidth but also speeds up inference. To address this, we introduce CARVQ, a post-training novel Corrective Adaptor combined with group Residual Vector Quantization. CARVQ relies on the composition of both linear and non-linear maps and mimics the original model embedding to compress to approximately 1.6 bits without requiring specialized hardware to support lower-bit storage. We test our method on pre-trained LLMs such as LLaMA-3.2-1B, LLaMA-3.2-3B, LLaMA-3.2-3B-Instruct, LLaMA-3.1-8B, Qwen2.5-7B, Qwen2.5-Math-7B and Phi-4, evaluating on common generative, discriminative, math and reasoning tasks. We show that in most cases, CARVQ can achieve lower average bitwidth-per-parameter while maintaining reasonable perplexity and accuracy compared to scalar quantization. Our contributions include a novel compression technique that is compatible with state-of-the-art transformer quantization methods and can be seamlessly integrated into any hardware supporting 4-bit memory to reduce the model’s memory footprint in memory-constrained devices. This work demonstrates a crucial step toward the efficient deployment of LLMs on edge devices.

## introduction transformer-based large language models (llm) are designed to handle extended contexts efficiently through processing tokens with attention mechanisms. transformer architecture can be primar- ## related works the rapid evolution of transformer-based architectures has significantly enhanced performance across natural language processing (nlp), computer vision (cv), and multimodal (mmmu) tasks. however, the massive computational and memory demands of these models remain a significant barrier to their deployment in resource-constrained environments such as edge devices and real-time applications. to address this issue, various compression techniques have been investigated. architecture preserving compression @xcite @xcite aims to reduce the size of transformer models while preserving their overall structure and operational flow. these methods focus on minimizing redundant computations or parameters without altering the architecture itself. quantization @xcite @xcite @xcite converts high-precision weights and activations into lower-precision representations to reduce memory and computational costs. by carefully balancing precision loss and performance, quantization methods are particularly effective in deploying models on hardware with constrained resources, such as system-on-chip (soc). another complementary approach, pruning @xcite @xcite , eliminates weights or attention heads deemed less impactful based on predefined criteria. for example, head pruning and structured pruning have shown that many weights in transformer blocks contribute minimally to the overall accuracy. these methods allow significant reductions in size and computational demand while retaining the architectural integrity of the model. architecture adaptive compression @xcite involves reconfiguring the model structure to achieve compression by replacing or simplifying specific layers. these methods embrace the notion that certain architectural modifications can provide significant efficiency gains while maintaining task accuracy. low-rank adaptation (lora) @xcite introduces lowrank parameter updates to large pre-trained weights, reducing memory footprint during fine-tuning on downstream tasks without the need to retrain. another innovative method, tensor train decomposition @xcite , decomposes large tensors into smaller low-rank tensors, significantly reducing memory footprint while maintaining accuracy. embedding layer compression @xcite specifically targets the reduction of parameters associated with embedding layers, which often constitute a substantial proportion of the overall model size, especially in multilingual architectures. in transformer models, embedding layers map input vocabulary and image-patch tokens to high-dimensional vectors, and their size grows proportionally with the vocabulary size. tensorgpt @xcite leverages tensor train decomposition to represent these highdimensional embeddings in a more compact manner, achieving parameter reduction with minimal degradation to model accuracy. however, the linear nature of tensorgpt results in accuracy drops in high-compression. similarly, dynamic vocabulary pruning @xcite adjusts the vocabulary size adaptively based on the task or data requirements, pruning infrequently used tokens to reduce the embedding matrix dimensions. however, these methods are limited by the requirement of fine-tuning, lacking generalization. ## carvq embedding compression in this section, we introduce a novel embedding carvq, defined as the composition of two methods: corrective adaptor (4.1) and group rvq (4.2). corrective adaptor works as non-linear intermediate maps σ 1 expanding the small dimension output of linear map σ 0 to the original embedding dimension. through this design we can efficiently compensate the precision loss from group rvq operation. figure 1 illustrates carvq's framework. ## conclusion in this work, we introduced carvq, a novel posttraining method for llm embedding layer compression to ∼1.6 bits without requiring specialized hardware to support lower-bit storage. by carefully reorganizing the matrix into vector-quantized tables, carvq reduces the embedding layer precision without utilizing equally low-bit datatypes. we focused on evaluating varying-complexity models on a diverse set of tasks, showing that carvq can achieve lower average bitwidth-per-parameter while maintaining reasonable perplexity and accuracy compared to scalar quantization. moreover, the carvq system can be seamlessly integrated into any hardware supporting 4-bit memory to reduce model memory footprint in significantly memory-constrained devices where every mb counts. carvq's compatibility with state-ofthe-art transformer quantization methods, which can be highly prone to error propagation, unveils promising potential for future scalability.