Paper: Training and inference of large language models using 8-bit floating point (ICLR-2024)

Reviewer comment (Experiments):
Do the authors think that their results will hold for FP8 training from scratch?

Author response:
Concerning the weakness about transparency and the first question, we understand how the term “training” can be confusing and have modified the title and introduction to better reflect the scope of the paper. Let us clarify that we don’t claim to have experiments for FP8 pre-training from scratch. However, from a conceptual point of view, our methodology can be applied to both pre-training and fine-tuning. That’s why we encapsulate both under the term “training”. In the experiments, we clarify that we are only doing fine-tuning (see the title of subsection 4.5 for instance). We would love to have pre-training results, but they are really expensive and only affordable for a handful of AI labs worldwide, especially for the models we tested of up to 13 billion parameters. As a result, we don’t include them here and leave them for future work.

Reviewer comment (Experiments):
Do the authors believe it's possible to use any of their methodology for other layers in the network such as attention?

Author response:
Concerning the second question about extending the FP8 methodology to other layers, we think so but need more time to have publishable results about it. As a result, we leave this question for future work.

Reviewer comment (Experiments):
The paper repeatedly emphasizes "training" in FP8 (e.g., in the title, in the abstract, etc.), yet I could not find any actual training experiments, that is training a large LLM from scratch, in the paper. The paper only performs some finetuning on GLUE tasks, which is significantly less interesting given that it is comparatively cheap and FP8 speedups thus not so crucial while, in many cases, even more affordable finetuning techniques like QLoRA also work well. I think significant presentation changes are required to clarify that the paper focuses on inference and finetuning.

Author response:
Concerning the weakness about “could not find any actual training experiments”, we understand how the term “training” can be confusing and have modified the title and introduction to better reflect the scope of the paper.  Let us clarify that we don’t claim to have experiments for FP8 pre-training from scratch. However, from a conceptual point of view, our methodology can be applied to both pre-training and fine-tuning. That’s why we encapsulate both under the term “training”. In the experiments, we clarify that we are only doing fine-tuning (see the title of subsection 4.5 for instance). We would love to have pre-training results, but they are really expensive and only affordable for a handful of AI labs worldwide, especially for the models we tested of up to 13 billion parameters. As a result, we don’t include them here and leave them for future work.

Reviewer comment (Novelty):
Most of the methodology appears to me like standard low quantized training techniques, e.g., using additional scales that are determined dynamically, adapted directly to FP8. Could you explain more precisely what exactly is new? I also did not find a Related Work section discussing this in more detail.

Author response:
Concerning the other two weaknesses about the novelty, the reviewer rightly points out that we take ideas from standard quantization techniques. For example, our max approach to choose the scales builds upon quantisation ideas from INT formats (see section 2.3). However, our paper is novel because we adapt the max approach to FP8 and detail its application to quantize weights, activations and gradients for both FP8 training and inference. Moreover, we shed light upon scaling decisions taken in popular FP8 software implementations (e.g. Transformer Engine).  Some of the fine-grained details and justifications of this implementation are not made explicit. We hope that by opening up our methodology and testing it in the experiments in Section 4, other FP8 researchers can build on top of it.The paper [5] that the reviewer mentions, claims to train large transformers in FP8 from scratch, but leaves many details of the implementation missing, hindering reproducibility. In contrast, we provide a pseudocode of the full forward and backward pass (see figure 1), explain the nuances between quantizing to FP8 the weights, activations and gradients, and show the mathematical formulas to compute the scaling biases (see subsection 2.3).

Reviewer comment (Novelty):
FP8 inference has been studied extensively by e.g. [3, 4] and also in the context of LLMs [1, 2]. Further, [5] finetunes (and even trains from scratch) large Transformers in FP8. Hence, the overall novelty of the work appears very low.

Author response:
Concerning the other two weaknesses about the novelty, the reviewer rightly points out that we take ideas from standard quantization techniques. For example, our max approach to choose the scales builds upon quantisation ideas from INT formats (see section 2.3). However, our paper is novel because we adapt the max approach to FP8 and detail its application to quantize weights, activations and gradients for both FP8 training and inference. Moreover, we shed light upon scaling decisions taken in popular FP8 software implementations (e.g. Transformer Engine).  Some of the fine-grained details and justifications of this implementation are not made explicit. We hope that by opening up our methodology and testing it in the experiments in Section 4, other FP8 researchers can build on top of it.The paper [5] that the reviewer mentions, claims to train large transformers in FP8 from scratch, but leaves many details of the implementation missing, hindering reproducibility. In contrast, we provide a pseudocode of the full forward and backward pass (see figure 1), explain the nuances between quantizing to FP8 the weights, activations and gradients, and show the mathematical formulas to compute the scaling biases (see subsection 2.3).

Reviewer comment (Evaluation):
What real-world inference speedups do you observe when finetuning and inferencing in FP8 using your setup?

Author response:
Lastly, concerning the question about real-world speedups, the hardware that we employ doesn’t have native FP8 native. This is mentioned in the hardware section H.2, in the appendix. As a result, we don’t claim to have such speedup numbers and leave it for future work.

Reviewer comment (Novelty):
The paper's contributions and novelty are not immediately clear. The methodology for calculating the Exponent Bias resembles the asymmetric quantization process for INT8, where the scaling factor is determined using a max operation. Furthermore, even within the scope of 8-bit floating point representation, determining the exponent bias based on the max operation has been explored in prior research. The author is recommended to clarify the paper's unique contributions, especially in comparison to the following studies:

[1] Tambe, Thierry, et al. "Algorithm-hardware co-design of adaptive floating-point encodings for resilient deep learning inference." 2020 57th ACM/IEEE Design Automation Conference (DAC). IEEE, 2020.

[2] Sun, Xiao, et al. "Hybrid 8-bit floating point (HFP8) training and inference for deep neural networks." Advances in Neural Information Processing Systems 32 (2019).

[3] Kuzmin, Andrey, et al. "Fp8 quantization: The power of the exponent." Advances in Neural Information Processing Systems 35 (2022): 14651-14662.

[4] Lee, Janghwan, and Jungwook Choi. "Optimizing Exponent Bias for Sub-8bit Floating-Point Inference of Fine-tuned Transformers." 2022 IEEE 4th International Conference on Artificial Intelligence Circuits and Systems (AICAS). IEEE, 2022.

Author response:
Concerning the weaknesses, the reviewer has caveats about our novelty and contribution. We acknowledge in the paper that the max approach to choose the scales builds upon quantisation ideas from INT formats (see section 2.3). However, our paper is novel because we adapt the max approach to FP8 and detail its application to quantize weights, activations and gradients for both FP8 training and inference. Moreover, we shed light upon scaling decisions taken in popular FP8 software implementations (e.g. Transformer Engine). The papers suggested by the reviewer have been important contributions for FP8, but in our view don’t cover: 1) details for practitioners to reproduce the FP8 methodology that has gained traction in the community, illustrated by the Transformer Engine library, Noune et al. [2022] and Micikevicius et al. [2022]; 2) training and inference validation for large language models of more than 1 billion parameters; 3) evolution of the scales during training and inference.

Reviewer comment (Experiments):
Comparisons with other numerical formats, such as INT8, block floating point, logarithmic number systems, and posit, are not discussed. For instance, the results in [5,6] indicate that INT8 performance is superior for inference, even for models like the Transformer.

[5] van Baalen, Mart, et al. "FP8 versus INT8 for efficient deep learning inference." arXiv preprint arXiv:2303.17951 (2023).  
[6] Zhang, Yijia, et al. "Integer or Floating Point? New Outlooks for Low-Bit Quantization on Large Language Models." arXiv preprint arXiv:2305.12356 (2023).

Author response:
Concerning the second weakness about lacking “comparisons with other numerical formats”, we refer the reader to our Appendix C, where we discuss them. It’s true that we don’t run experiments with those other formats, but we believe that our paper is self-contained with the focus on FP8. Other papers cited in our manuscript already cover those comparisons, see for instance Kuzmin et al. [2022] and van Baalen et al. [2023].

Reviewer comment (Theory):
What is the reason for using the max operation to compute the exponent bias? Why didn't the author consider other statistical metrics such as mean or mode? The max operation typically works best when the distribution is symmetric. However, the distribution in deep learning is often asymmetric.

Author response:
Regarding the question about why using the max operation and not others, we focus on max since it is simpler than computing the mode or mean while being sufficient to match FP8 training and inference validation compared to higher precision (see section 3). Moreover, max is the methodology gaining traction in FP8 libraries like Transformer Engine. Other papers that we cite already cover those comparisons, see Micikevicius et al. [2022] and van Baalen et al. [2023] which employ other methods like MSE or percentile.