GSQ -Tuning: Group-Shared Exponents Integer in Fully Quantized Training for LLM s On-Device Fine-tuning (2025)

Abstract: Large Language Models (LLMs) fine-tuning technologies have achieved remarkable results. However, traditional LLM fine-tuning approaches face significant challenges: they require large Floating Point(FP) computation, raising privacy concerns when handling sensitive data, and are impractical for resource-constrained edge devices. While Parameter-Efficient Fine-Tuning (PEFT) techniques reduce trainable parameters, their reliance on floating-point arithmetic creates fundamental incompatibilities with edge hardware. In this work, we introduce a novel framework for on-device LLM fine-tuning that eliminates the need for floating-point operations in both inference and training, named GSQ-Tuning. At its core is the Group-Shared Exponents Integer format, which efficiently represents model parameters in integer format using shared exponents among parameter groups. When combined with LoRA-like adapters, this enables fully integer-based fine-tuning that is both memory and compute efficient. We demonstrate that our approach achieves accuracy comparable to FP16-based fine-tuning while significantly reducing memory usage ( 50%). Moreover, compared to FP8, at comparable performance levels, our method can reduce 5x power consumption and 11x chip area, making large-scale model adaptation feasible on edge devices.

## introduction recent advances in large language models (llms) have delivered impressive results in a variety of natural language tasks @xcite . llms are typically trained in several stages, including large-scale pretraining followed by one or more fine-tuning phases @xcite . llm fine-tuning approaches like supervised fine-tuning † equal contribution ‡ work done as an intern at houmo ai (sft) @xcite , usually employ curated, high-quality corpora for refining the model with a standard language modeling @xcite . despite their effectiveness, most llm finetuning approaches require powerful cloud servers or gpus equipped with large memory capacities. this poses two significant challenges in real-world settings: (1) uploading sensitive data to remote servers poses a fundamental privacy risk, and (2) in many practical scenarios, models must be deployed on resource-constrained edge devices-such as mobile processors or embedded ai accelerators-where memory and power budgets are tightly limited. such constraints become critical in llm's personalized applications, where data cannot be shared with the cloud and model updates must remain local to ensure privacy. meeting these challenges thus necessitates on-device adaptation methods capable of preserving data privacy and functioning within the limited memory and compute budgets of edge hardware. parameter-efficient fine-tuning (peft) @xcite techniques such as lora @xcite and qlora @xcite alleviate part of this burden by reducing trainable parameters to around 1% of the original model. unfortunately, they remain reliant on floating-point operations for both forward and backward passes, which clashes with edge-device constraints in three ways. first, during fine-tuning, weights, activation and gradients must be stored and updated in high-precision floating-point. it introduces additional overhead or even makes the llm fine-tuning impractical on edge devices. second, floating-point representations incur high memory overhead (e.g., fp16 doubles the memory cost compared to int8; for a 7b-parameter model, this can surpass 20gb memory during fine-tuning process, presenting substantial challenges for mobile processors). last but not least, commercial edge ai accelerators (e.g., qualcomm hexagon (qualcomm, 2024)) typically get peak throughput only on integers, leaving up to 84% of compute units idle under fp16 training. therefore, eliminating floating-point arithmetic for fine-tuning would have a substantial impact on software, hardware, and application design for efficient on-device llm adaptation (arm, 2020; @xcite . while previous studies on integer quantization @xcite @xcite verify the feasibility of inference, they do not extend to gradient quantization, which is required for effective finetuning of llms at the edge. in this paper, we propose a new framework for resource-efficient on-device llm fine-tuning, termed gsq-tuning. central to our method is the group-shared exponents integer format, a novel quantization strategy that replaces floating-point with a specialized integer-based representation. we integrate this with parameter-efficient lora-like modules to enable fully on-device fine-tuning without incurring large memory and computation costs. we further examine this design through a pareto frontier analysis, which demonstrates how various bits-rank settings impact the trade-off between fine-tuning memory costs and accuracy. extensive experiments across models of varying scales, different fine-tuning datasets, and diverse tasks have demonstrated the effectiveness and generalizability. we highlight our main contributions as follows: ## method in this section, we present gsq-tuning, a fully quantized training method for on-device llm fine-tuning. we begin by reviewing the fundamentals of llm peft, ## experiments foundation models and evaluation metrics. we apply our method to the entire llama family, including llama-2 (7b/13b/70b) @xcite , and llama-3 (3b-8b). we evaluate the fine-tuning models on up to 9 zeroshot commonsense question-answering (csqa) tasks using the lm-evaluation-harness (version 0.4.7) @xcite , including boolq @xcite , hellaswag @xcite , lambada (openai) @xcite , open-bookqa @xcite , piqa @xcite , siqa @xcite , wino-grande @xcite , arc-easy, and arc-challenge @xcite . the finetuning dataset follows alpaca @xcite , with 52k instruction data from text-davinci-003. besides, we also report the memory cost (mem.(g)) during the model fine-tuning process. training details. we employ the whole finetuning process based on llama-factory @xcite . we implement gsq-tuning in pytorch using models from hugging face. we freeze the parameters of the linear modules and update a smaller (low-rank) set of parameters during the fine-tuning. the group size of our gsq-tuning is 32. we fine-tune models using the 8-bits adamw optimizer @xcite in bfloat16 precision. we choose the constant learning rate schedule and set the learning rate to be 1 × 10 -5 for all models. in all cases, we tune the hyperparameters on the base bf16 tasks, and re-use the same values for low-precision training. we always perform single-epoch experiments using a linear learning rate warm-up of 100 steps. the batch size and sequence length are fixed at 16 and 2048. the number of fine-tuning steps is 3.24k for alpaca. ## related work parameter-efficient fine-tuning (peft). peft reduces memory and computational costs by introducing a small set of trainable parameters while keeping the pretrained model frozen. approaches including soft prompt tuning @xcite , partial fine-tuning @xcite , and low-rank adaptation @xcite . among these, lora stands out as a seminal work, injecting trainable low-rank matrices into linear layers to enable efficient fine-tuning without modifying the base model weights. qlora extends this with 4-bit nf4 quantization and double quantization, supporting 65b model fine-tuning on a single 48gb gpu with minimal performance degradation. recent work further improves quantization-aware fine-tuning @xcite and extends lora for better efficiency, stability, and performance @xcite @xcite @xcite . quantization. much works @xcite @xcite make efforts to accelerate the llms. for instance, gptq @xcite quantizes weights to 3-4 bit with slight accuracy drop based on approximate second-order information. awq @xcite and smoothquant @xcite explore the scheme of smoothing by detecting the importance of different activation channels. recent works (e.g., quarot (ashkboos et al., 2024), spinquant @xcite ) further suppress outliers by utilizing computation-invariant rotation transformation. however, above methods focus on the inference optimization. leveraging the key benefits of fqt, several studies @xcite @xcite @xcite have explored its implementation and optimization. however, challenges remain, especially during the backward pass due to the wide dynamic range of gradients. for example, lm-fp8 @xcite trains llms from scratch using fp8, achieving performance comparable to bf16. some studies have also explored full integer quantization, utilizing efficient hardware implementations. switchback @xcite quantizes partial matrix multiplications with int8, but it is limited to vision models with up to 1b parameters. jetfire @xcite proposes a 2d block-wise quantization approach that maintains accuracy and achieves significant memory savings (1.4-1.5x) when training in int8. however, these methods have not yet been explored to fine-tuning tasks, and using fqt to lower-bit (< int8) poses considerable challenges. ## conclusion in this paper, we propose gsq-tuning, a resourceefficient framework that addresses the critical challenges of floating-point dependency, privacy risks, and hardware incompatibility in on-device llm fine-tuning. by integrating group-shared exponents integer (gse) quantization with parameter-efficient adaptation, our method achieves three key advancements:(1) full integer pipeline: eliminates floating-point operations across both forward and backward passes, reducing memory usage by 50% compared to fp16 while maintaining comparable accuracy. (2) hardware-optimized design: the gse format reduces metadata overhead via groupwise exponent sharing, enabling 5-8bit integer representations. combined with lora-like adapters, this achieves 5 × lower power consumption and 11× smaller chip area compared to fp8 at equivalent accuracy levels. (3) practical deployment guidance: a pareto frontier analysis guides optimal bit-rank configurations for diverse edge constraints. these innovations establish gsq-tuning as a foundational step toward democratizing llm adaptation for resource-constrained environments. this breakthrough makes private, on-device llm adaptation practical for sensitive applications. future work will explore sub-4bit quantization to further push the boundaries of edge ai.