Paper: KVTQ: Compressing the KV Cache to Hardware Efficient Ternary Digits by Fine-Grained Dynamic Quantization (ICLR-2024)

Reviewer comment (Experiments):
The authors present statistics regarding the maximum and minimum values of the KV cache in each layer, but do not delve into the distribution of data within each KV cache. While data distribution may be less relevant when quantizing data into n-bit integers, this paper opts for ternary digit quantization. Consider a key embedding to follow a normal distribution with the distance between the average and maximum value being 3 times the standard deviation (Max = $\mu + 3\sigma$). According to the authors' equation ($X_q = \lceil \frac{X}{\Delta} \rfloor, \Delta = \text{max}(|X|)$, approximately 86.6% of the numbers will be quantized to zero ($P(-1.5<\frac{x-\mu}{\sigma}<1.5)$). Such a substantial portion of key embeddings being quantized to zero might suggest sparsity in attention scores. Hence, it would be more persuasive if the authors compared their work against models employing sparse attention, as opposed to traditional attention mechanisms.

Author response:
As show in table 3, we evaluate the sparse ratio of the KV cache of difference series of 
models under our KVTQ quantization setting.
The KV cache of different models has different sparse ratio after quantization.
The sparse ratio of the KV cache with the largest quantization step is high as your 
analyze.
But if we only use one channel of ternary digit to express the KV cache, the accuracy is 
unacceptable.
So we use 4 channels of ternary digits with different quantization step to express the K 
cache and 3 channels for the V cache.
The sparse ratio of the KV cache with the smallest quantization step is about 50%.


Scissorhands [1] compresses the KV cache by only saving the key-value embeddings with 
high attention scores, which is one kind of sparsity.
They give the perplexity result as a curve in a figure. 
It can be seen from the figure that when the sparse ratio is greater than 50%, ppl 
increases significantly.

Reviewer comment (Experiments):
Would the utilization of a sparse attention method yield superior results compared to KVTQ in terms of perplexity, runtime, and memory usage?

Author response:
As show in table 3, we evaluate the sparse ratio of the KV cache of difference series of 
models under our KVTQ quantization setting.
The KV cache of different models has different sparse ratio after quantization.
The sparse ratio of the KV cache with the largest quantization step is high as your 
analyze.
But if we only use one channel of ternary digit to express the KV cache, the accuracy is 
unacceptable.
So we use 4 channels of ternary digits with different quantization step to express the K 
cache and 3 channels for the V cache.
The sparse ratio of the KV cache with the smallest quantization step is about 50%.


Scissorhands [1] compresses the KV cache by only saving the key-value embeddings with 
high attention scores, which is one kind of sparsity.
They give the perplexity result as a curve in a figure. 
It can be seen from the figure that when the sparse ratio is greater than 50%, ppl 
increases significantly.

Reviewer comment (Evaluation):
While ternary computations can simplify attention-related calculations, the authors haven't quantified the reduction in latency or the number of FLOPs saved by their method.

Author response:
**Flops(item 1)**

In section 3.5 we describe the amount of multiplication operations in the attention 
block, but due to limited space we did not give a complete example.
We will refine this part and add more results in appendix.

Here we use LlaMa-7B as an example:

For LlaMa-7B the head number is 32 and head size is 128.
If we assume the input sequence is 1024 and the maximum sequence length is 4096 and the 
average sequence will be $(1024 + 4096) / 2$.
As mentioned in section3.5, the KVTQ method reduce 
$4 \times B \times NH \times S \times HS$ times of multiplication operations.

If we assume the batch size is 16, in the generation process the reduced multiplication 
operations of one layer of the LlaMa-7B  model are:
$$ 4 * 16 * 32 * (1024 + 4096) / 2 * 128 * (4096 - 1024) = 1.875T $$
For the LlaMa-7B model with 32 transformer blocks, the total reduced multiplication 
operations are:
$$ 1.875T * 32 = 60T $$

Reviewer comment (Evaluation):
Given that ternary representation uses 2 bits to represent -1, 0, or +1, its memory footprint might surpass binary-based quantization. A comparison of memory usage between previous quantization methods and the proposed approach is essential.

Author response:
**Memory usage(item 2)**

In KVTQ, we use 4 groups of ternary digits to express the K cache and 3 groups of ternary
digits to express the V cache.
As you said, if we want to storage the ternary K cache in binary memory device directly, 
we have to use 8 bits and for V cache, we have to use 6 bits.
The memory usage is less than the 8-bit static quantization used in TRT-LLM but the 
memory usage is higher than the K4/V4 fine-grained dynamic quantization we proposed in 
our paper as comparison method.

Ternary digits are more suitable to be stored in ternary memory device.
And saving memory is not the only target considered by the KVTQ method. 
The K4/V4 method we proposed in our paper saves more memory compared to the KVTQ method, 
but the K4/V4 method does not have the computing advantages which we mentioned in section
3.5.

Reviewer comment (Evaluation):
The correlation between sparsity and computational reduction doesn't directly equate to reduced latency or improved throughput. Instead of merely highlighting channel sparsity, tangible hardware benefits should be assessed.

Author response:
**Sparsity and performance(item 3)**

As you said, sparsity and computational reduction doesn't directly equate to reduced 
latency or improved throughput.
On existing LLM inference devices, such as GPUs, the KVTQ method can be run.
But the KVTQ method requires a customized ASIC to release its advantages.
For hardware such as GPU, the K4/V4 method we proposed in our paper is more suitable.

Reviewer comment (Writing):
The paper's description of the actual quantization method might lead to misleading. The term "ternary" suggests that K and V are genuinely ternary with values {-1, 0, +1}. However, the paper assigns "multiple channels" to each value with varying steps. This essentially equates to a higher quantization bit count. As the paper states, “we use 4 channels of ternary digits for the key embeddings and 3 channels of ternary digits for the value embeddings.” This means K is 4-bit quantized, and V is 3-bit quantized? This crucial point lacks adequate discussion and might mislead readers.

Author response:
**Why we use "ternary"(Q1)**

The ternary digits have two nice properties.
First for $A \times B $, when either A or B contains only ternary digits, the matrix 
multiplication can be completed by addition and subtraction operations.
4 bit/3 bit integers do not have this feature.
We give a tiny example in figure 3.

Second the ternary representation will result in sparsity.
We can take advantage of this feature to reduce the amount of calculations.
4 bit/3 bit integers do not have this feature.
Figure 3c is an example.

For these two reasons, we use “ternary” in our paper.

Reviewer comment (Evaluation):
The quantization method utilized is dynamic, meaning the quantization step must be dynamically determined. This approach may not be hardware-friendly. To determine a single max and min value requires scanning the entire tensor. For larger tensors, this method could introduce significant latency.

Author response:
**Overhead of dynamic quantization(Q2)**


For text generation applications, the KV cache is quantized once and used multiple times. 
During the generation process, for each new query, we only have to quantize the key-value
embeddings corresponding the current query since the key-value embeddings of the past 
querys have been cached.
For each new query, the size of key-value embedding is $1 * hiddensize$.

Compared to static quantization of the KV cache, the only difference is that we have to
compute the quantization step during the inference process for dynamic quantization.
The amount of calculation introduced by dynamic quantization is relatively small and can
be ignored.

Reviewer comment (Novelty):
The ternary compensation algorithm employed originates from ABC-Net and is not original.

Author response:
**Relationship with ABC-Net(Q3)**

Ternary quantization have been used in computer vision tasks formerly and we are the 
first which use this technique in quantizing the KV cache of LLMs.
Previously, binary/ternary quantization was used within the scope of quantization-aware 
training(QAT). 
Our KVTQ applies ternary quantization under the scope of post-training quantization(PTQ).

We mentioned ABC-Net for it gives us the idea to use the **multiple groups** idea rather 
than it use ternary quantization. 
ABC-Net try to quantize AI models of computer vision tasks to binary value.
ABC-Net focuses on binary quantization and the method is under the QAT scope. 
We will refine the description of the relation between our work and ABC-Net.

Reviewer comment (Writing):
Can you provide clarity on the choice of using "ternary" in your terminology when the actual quantization might suggest higher bit counts?

Author response:
**Why we use "ternary"(Q1)**

The ternary digits have two nice properties.
First for $A \times B $, when either A or B contains only ternary digits, the matrix 
multiplication can be completed by addition and subtraction operations.
4 bit/3 bit integers do not have this feature.
We give a tiny example in figure 3.

Second the ternary representation will result in sparsity.
We can take advantage of this feature to reduce the amount of calculations.
4 bit/3 bit integers do not have this feature.
Figure 3c is an example.

For these two reasons, we use “ternary” in our paper.

Reviewer comment (Novelty):
Since the ternary compensation algorithm is taken from ABC-Net, how is your quantization method different from it?

Author response:
**Relationship with ABC-Net(Q3)**

Ternary quantization have been used in computer vision tasks formerly and we are the 
first which use this technique in quantizing the KV cache of LLMs.
Previously, binary/ternary quantization was used within the scope of quantization-aware 
training(QAT). 
Our KVTQ applies ternary quantization under the scope of post-training quantization(PTQ).

We mentioned ABC-Net for it gives us the idea to use the **multiple groups** idea rather 
than it use ternary quantization. 
ABC-Net try to quantize AI models of computer vision tasks to binary value.
ABC-Net focuses on binary quantization and the method is under the QAT scope. 
We will refine the description of the relation between our work and ABC-Net.

Reviewer comment (Novelty):
The novelty of the proposed ternary quantization is limited since it was first proposed by ABC-Net.

Author response:
**Relationship with ABC-Net(item 1)**

Ternary quantization have been used in computer vision tasks formerly and we are the 
first which use this technique in quantizing the KV cache of LLMs.
Previously, binary/ternary quantization was used within the scope of quantization-aware 
training(QAT). 
Our KVTQ applies ternary quantization under the scope of post-training quantization(PTQ).

We mentioned ABC-Net for it gives us the idea to use the **multiple groups** idea rather 
than it use ternary quantization. 
ABC-Net try to quantize AI models of computer vision tasks to binary value.
ABC-Net focuses on binary quantization and the method is under the QAT scope.
We will refine the description of the relation between our work and ABC-Net.

Reviewer comment (Evaluation):
This paper lacks measured memory usage and memory footprint of the KV cache for the proposed KVTQ method.

Author response:
**Memory usage(item 2)**

In KVTQ, we use 4 groups of ternary digits to express the K cache and 3 groups of ternary
digits to express the V cache.
If we want to store the ternary K cache in binary memory device directly, we have to use 
8 bits and for V cache, we have to use 6 bits.
Ternary digits are more suitable to be stored in ternary memory device.

Saving memory is not the only target considered by the KVTQ method. 
The K4/V4 method we proposed in our paper saves more memory compared to the KVTQ method, 
but the K4/V4 method does not have the computing advantages which we mentioned in section
3.5.