Continual Quantization-Aware Pre-Training: When to transition from 16-bit to 1.58-bit pre-training for B it N et language models? (2025)

Abstract: Large language models (LLMs) require immense resources for training and inference. Quantization, a technique that reduces the precision of model parameters, offers a promising solution for improving LLM efficiency and sustainability. While post-training quantization methods typically achieve 4-8 bits per parameter, recent research suggests that training LLMs with 1.58 bits per weight parameter from scratch can maintain model accuracy while greatly reducing memory requirements and energy consumption at inference time. Here, we investigate a training strategy for quantization-aware pre-training, where the models are first trained with 16-bit precision and then transition into 1.58-bit quantization-aware training. Our results on 11 downstream tasks, show that this 16-to-1.58-bit training strategy is preferable over full 1.58-bit training and leaves models closer to those which have undergone 16-bit training. We further investigate the effects of retaining the optimizer state at the transition point and gradually phasing in quantization strength - finding that both techniques alleviate the magnitude of loss spikes, but also that these effects can be compensated through further training.

## introduction large language models (llms) have revolutionized natural language processing and are making a strong entry into both related and unrelated industries. however, deployment of llms comes with a series of obstacles such as memory-usage, latency, and throughput. environmental considerations regarding energy consumption of both training and inference becomes an increasingly important aspect @xcite . quantization-aware training of language models, i.e. preparing the model for later quantization already during training and thereby preventing performance degradation between training and inference, has shown promising results @xcite . however, the resulting models tend to require more parameters to compensate for the reduction in bit-precision per parameter @xcite . here, we investigate a strategy of first training with standard precision in an initial phase, followed by a second phase of 1.58-bit quantization-aware training with weights quantized to either -1, 0, or 1. reducing inference compute demands of language models is particularly critical in the context of recent advances in scaling test time compute @xcite @xcite . if the trend of scaling test-time compute continues, we can assume that inference will soon dominate the resource consumption in a language model's life cycle. recent works in 1-bit @xcite ) and 1.58-bit quantization-aware training @xcite @xcite , demonstrate the potential of training in 1.58-bit precision while retaining most of the performance, mitigating some of the drawback of existing post-training quantization techniques, such as performance degradation in both nlp and computer vision @xcite . to achieve competitive model performance, these quantization-aware training strategies keep 16-bit-precision weights at training time ("shadow weights"), which are quantized on-the-fly to 1bit or 1.58-bit precision during forward passes. straight-through estimated gradients @xcite are then used to update the shadow weights. while such a training strategy initially requires more memory and compute than pre-training a regular language model, its benefits can be harvested after training. at inference time, the final shadow weights can be quantized once and for all, after which the shadow weights can be discarded, yielding a model with 4-5 times reduced memory footprint, compared to 16-bit model. with tailored kernels, this would allows us to replace costly matrix multiplications within linear layers with computationally more efficient integer addition operations. although these methods provide an exciting avenue for efficient inference on models trained with 1.58-bit quantization-aware training techniques, existing models are not eligible for efficient inference out of the box @xcite . this is increasingly important as increasing model size implies corresponding resource requirements, making training from scratch both resource heavy and environmentally challenging. this calls for an investigation of a resource efficient strategy. recent analysis have hinted at 7-8 bits being compute optimal @xcite . pushing the bit-representation even lower currently requires a quantization-aware training strategy as proposed in bitnet @xcite . concurrent work on quantization-aware training has shown that it is generally possible to convert a 16-bit model and into a 1.58-bit model through continual pre-training @xcite . however, the experiments reported upon only consider the case of starting with an already pre-trained language model and do not compare against training with low precision from scratch. in related matter, recent work has shown that "overtrained" networks, such as fully-pre-trained language models, do not represent an ideal starting point for quantization @xcite . it is so far unclear which strategy to choose when pre-training from scratch: is it preferable to train with 1.58-bit quantization-aware training all the way, or is it preferable to start with standard 16-bit training and then transition into 1.58-bit quantization-aware training (see figure 1 ). and if so, when should we switch from 16-bit to 1.58-bit training? is a it beneficial to gradually phase in the quantization strength? and what role does the retention of the optimizer state (if available) play? in this work, we investigate these questions to shed new light on the potential of continuing a pre-training run in 1.58-bit. we aim to provide insights into the potential of converting existing pre-trained models to 1.58-bits and into developing more efficient ways of pre-training new models. ultimately, this work contributes to a future with more efficient, environmentally-friendly llms and also allows to convert existing 16-bit models into 1.58-bit models, even in low-resource settings. in order to answer these questions and achieve our aims, we systematically perform a number of experiments to identify a data-optimal transition point from 16-bit to 1.58-bit training. we further investigate whether it makes sense to phase in the quantization strength and whether the optimizer state is of importance, providing a basis for future work on expand upon and for practitioners to receive guidance in converting existing models and pre-training new models. challenging both conventional wisdom and common-sense expectations, our systematic comparison reveals that models that are first trained with 16-bit precision, and only later transition into 1.58-bit quantization-aware training, are more ultimately effective than models that are fully trained with 1.58-bit quantization-aware training. in addition, we find that, while retaining the optimizer state indeed helps to alleviate a temporary increase in training loss, continual pre-training with a fresh optimizer state attains similar loss values after a limited number of optimization steps. furthermore, we find that gradually introducing quantization strength provides little to no benefits. a continual pre-training with full quantization strength temporarily increases training loss, but attains similar loss values, and also downstream performance, after a limited number of steps. these results together indicate that the availability of optimizer state is not an obstacle when continuing pre-training from openly available language models and that no new hyperparameters (regarding phasing in quantization strength) are needed, enabling an easier transition of existing pre-trained models, with smaller amounts of data. in summary, our contributions are as follows: ## related work early attempts of ternary-weight neural nets. the exploration into ternary weights was motivated by finding a better trade-off between accuracy and complexity than binary neural networks, which had suffered a substantial decrease in performance, effectively hindering the usability of such networks @xcite . earlier attempts of binary-and ternary-weight neural networks showcased ternary superiority over binary weights, while showing promising results in the computer vision domain employing a direct optimization of the quantization @xcite . post-training quantization. the most common approaches for quantization fall under the category of post-training quantization. those include approaches such as generative pre-trained transformer quantization (gptq) @xcite and activation-aware weight quantization (awq) @xcite . a similar proposal for the vision transformer @xcite represents one of many efforts in the computer vision domain. posttraining quantization approaches come with an inherent decrease in performance, trading increased latency and throughput and decrease memory for precision @xcite . quantization-aware training. quantizationaware training was already proposed in earlier work on post-training such as llm-qat @xcite and qa-lora @xcite . these methods directly optimize the quantized weights with respect to an objective function such that there is no decrease in performance when the model is used for inference. recently, we have seen a number of works on 1-bit @xcite and 1.58-bit @xcite quantization-aware techniques demonstrating strong performance in llm performance, yielding a small or no loss in precision depending on model sizes. other works have demonstrated strong potential for multi-modal architectures @xcite and spiking language models @xcite . lastly, we have seen an investigation into the potential of 1.58-bit in small language and vision models and the definition of a scaling law for decoder-only models for reintroducing capacity @xcite . similar scaling laws hold for encoder-only models while encoder-decoder models seem to be less predictable @xcite . this latest work further shows that also non-transformer models, such as plain multi-layer perceptions and graph neural networks can attain similar performance as their 16/32bit counterparts, even without increasing the number of parameters. ## methods we first recapitulate the basics of quantizationaware training (section 3.1) before describing the proposed strategy of continual 1.58-bit pre-training (section 3.2) and discussing critical considerations concerning optimizer states (section 3.3) and gradually phasing in quantization strength (section 3.4). ## experimental setup we conducted experiments investigating the potential of continuing the pretraining in 1.58-bits from (partially) pre-trained 16-bit models, initializing the 1.58-bit model's "shadow-weights" with the pre-trained 16-bit weights. furthermore, we investigate the impact of optimizer state retention and phasing in quantization strength. architecture specifically, we employ the model architecture of open language models @xcite in their official 1b parameters configuration. we use the bitlinear package 1 to all replace all nn.linear layers within an olmo model by bitlinear layers (as described in section 3), which includes both projection layers within the attention and feed-forward modules. training we train all models on the dolma dataset @xcite , with olmo's standard hyperparameters foot_1 . in particular, optimization is carried out by the adamw optimizer (kingma and ba, 2015) with a cosine learning rate scheduler with warmup. we employ the same learning rates for both the 16 and 1.58-bit baselines in accordance with observations in our prior work @xcite . we employ a sequence length of 2048 and a batch size of 2048, totaling to a batch size of 4m tokens. each experiment is run for a total of 10,000 optimizer steps. ## results we first present the results regarding 16-to-1.58bit quantization-aware training in section 5.1, followed by results comparing effects of and optimizer state retention (section 5.2) and of phasing in of quantization strength (section 5.3). lastly, we report the results of the evaluation on downstream tasks in section 5.4. ## discussion through a comprehensive set of experiments where we have full control over the pre-training data and regimen, we have shown that, counterintuitively, quantization-aware training of 1.58-bit models from scratch is neither ideal nor most efficient for obtaining the best possible 1.58-bit models. in addition, we find that previously proposed strategies of gradually phasing in quantization strength are not needed as long as there are sufficient optimization steps to catch up. furthermore, we make similar observations for the optimizer states: resetting the optimizer results in a spike in training loss, but the loss can be recovered after relatively few further optimization steps. crucially, our experiments on downstream tasks indicate that continual 16-to-1.58-bit pre-training is a highly effective training strategy, consistently outperforming full 1.58-bit training, and sometimes even exceeding downstream performance of 16-bit models. our results show that a period of standard training at higher precision (16-bit), and then continuing the pre-training with quantization-aware 1.58-bit training constitutes a preferable training regimen. we hypothesize that this is because of particular challenges that 1.58-bit quantization-aware training imposes on optimization. adjusting the weights is more difficult under 1.58-bit training, because the quantized weight may very well remain stable even though the "shadow weights" change and their gradient is estimated straight-through. thus, starting from random parameters, as opposed to a set of parameters obtained through 16-bit training, seems to be more challenging to optimize. interestingly, the continual pre-training variants with an initial period of standard training have led to lower over-all training loss compared to 1.58-bit training from the start, even when being restricted to the same amount of data and number of steps. re-initializing the optimizer when transitioning to 1.58 bit training yields spikes in the loss-curves caused by the loss of the momentum terms -both with and without quantization phase-in. retaining the optimizer states or phasing in quantization strength dampens this effect substantially. notably, without phased-in quantization strength, the spikes have a much higher magnitude. however, these effects are dampened after a few optimization steps and the tweaks to mitigate the spikes do not yield a gain in final performance. we hypothesize that when retaining the optimizer state, the optimizer still needs to react to the coarseness of the 1.58 bit representations as well as the new type of feedback though the straight-through estimator, giving a similar yet shorter lasting spikes. in the end, if there is sufficient data available for a moderate amount of optimization steps, then it is not necessary to avoid the loss spikes. this finding has important implications: if one aims for the best possible, most efficiently trained, 1.58-bit model given a fixed amount of data, one should still first train a 16-bit model on a proportion of the data (e.g., between 20 and 40%) and only then introduce quantization into the training. keeping the optimizer state is beneficial to avoid loss spikes, but if the optimizer state is not available, sufficient further optimization steps enable the model to recover, eliminating the need for optimizer checkpoints -which is crucial for continual 1.58-bit pre-training of existing models. alleviating inference compute demands comes with immediate gains for democratization of ai research @xcite and for advances in scaling test-time compute. from a different perspective, our results further support the idea of 1.58 continual pre-training be-(a) arc-easy @xcite (b) rte @xcite (c) copa @xcite (d) hellaswag @xcite ) (e) mrpc @xcite (f) openbookqa @xcite (g) piqa @xcite ing applied to fully pre-trained base models -opening avenues of research to convert arbitrary 16bit pre-trained models into corresponding 1.58-bit models. determining the minimum amount of data required for this conversion is an interesting direction for future work. unlike post-quantization methods, quantization-aware training schemes enable lossless quantization (considering the difference training and inference performance). this is particularly important in safety-critical domains (e.g., medical), where "blind" post-training quantization could lead to an unexpected degradation of performance. the deployment of 1.58 models requires less memory, due to the fact that 5 ternary weights and be packed into one 8-bit integer (couture-harpin, 2024). naturally, this will also speed up the hardware load-times with a small overhead of unpacking the weights online, during inference. the computational load is also reduced by the replacement of multiplication and subsequent addition with addition alone. however, fully realizing these benefits in practice requires more specialized hardware implementation, which is already an activate area @xcite . future hardware generations designed to include support for this scheme will further enhance efficiency we further demonstrate that continually pretrained 1.58-bit models are competitive on a broad range of downstream tasks, showcasing the capabilities of these inference-efficient models. notably, the number of parameters was kept fixed between conditions in all our experiments, indicating that those continual 1.58-bit pre-training goes beyond recently proposed scaling laws for quantized networks in general @xcite and bitnet-style quantization-aware training @xcite in particular. however, an analysis of the scaling behavior of our proposed continual quantization-aware pre-training strategy is left for future work. a further promising direction for future work would be to study the effect of 1.58-bit instruction finetuning, e.g., whether instruction tuning data might be sufficient for converting existing base models from 16 to 1.58 bits. ## conclusion we conducted a systematic comparison between language models trained from scratch with 1.58-bit precision and 16-to-1.58-bit continual pre-training paradigms. our results show the existence of a data-optimal transition point for switching from 16-bit to 1.58-bit training, providing insights into efficient low-bit training strategies. we evaluated the downstream performance of 1.58-bit models, highlighting their viability for real-world applications. these findings contribute to the broader discussion on low-bit training efficiency and its implications for scalable ai model development. future work may determine the minimal amount of data needed to successfully convert a 16-bit model into a 1.58bit model through continual pre-training.