Paper: Efficient and Quantization-Friendly Ternary Fourier Convolution Algorithms (ICLR-2024)

Reviewer comment (Experiments):
It seems to lack qualification results to support Table 2 such as comparisons of feature maps.

Author response:
**Q1**: It seems to lack qualification results to support Table 2 such as comparisons of feature maps.

**A1**: Since we opt for a classification task on Imagenet dataset in the comparison experiments, the model accuracy serves as an indicator of the algorithm's quality. For example, using our proposed SFC6(7×7, 3×3) algorithm under Int8 arithmetic, the accuracy reduction is to only 0.15%, which substantially outperforms similar fast convolution quantization work.

1. **Absolution studies on feature map.**

 In the **Appendix A.2**, we plot the layer-wise Mean Squared Error (MSE) error of feature maps as the qualification metric to compared different algorithms, using AdaQuant[1] as the fine-tune method. Some of its contents are shown in Table R3:

**Table R3. MSE error of feature maps for different algorithms**

| Resnet18,conv4.1.2    | Standard convolution |        | Winograd (4×4, 3×3) |      | SFC6 (7×7, 3×3) |        |
| --------------------- | -------------------- | ------ | ------------------- | ---- | --------------- | ------ |
| is fine-tune          | No                   | Yes    | No                  | Yes  | No              |        |
| MSE error to FP model | 2.7e-3               | 2.8e-4 | 1.3                 | 0.36 | 2.3e-2          | 6.8e-3 |

We can see that, in our proposed SFC algorithm, **the MSE loss of feature maps is significantly lower than that in the Winograd algorithm**. However, it is worth noting that, when using the scaling-factor fine-tune method, the MSE loss of feature maps in the SFC algorithm is smaller than that in the standard convolution, while the accuracy of the quantized model does not improve. It enlightens us that the MSE error of the feature map is not the sole criterion to judge the quantization quality.

Reviewer comment (Experiments):
The ablation study on scale factor (e.q. 14) seems to be needed for its effectiveness.

Author response:
**Q2**: The ablation study on scale factor (e.q. 14) seems to be need for its effectiveness.

**A2**: We add new ablation experiments on the scaling factor. The detailed results are given in **Table R1-2**. 

**Table R1**

| Resnet18(69.76%) | Minmax | Scaling Factor Fine-tune |
| ---------------- | ------ | ------------------------ |
| Accuracy         | 67.42  | 69.58                    |

We can see by involving scaling factor fine-tune scheme, the model accuracy is better preserved.

We also add ablation experiments on the grain of scaling factor. A fine-grain scaling factor can improve the accuracy of quantized models, but it may affect the deployment efficiency on hardware. For activation, frequency-wise quantization is necessary. For filters, channel-wise and frequency-wise yield similar results, both outperforming tensor-wise. Where hardware permits, simultaneous frequency-wise and channel-wise quantization would give the best results.

**Table R2**

| scaling factor grain of Activations | Tensor-wise  | Frequency-wise | Frequency-wise | Frequency-wise              |
| ----------------------------------- | ------------ | -------------- | -------------- | --------------------------- |
| scaling factor grain of filters     | Channel-wise | Channel-wise   | Frequency-wise | Frequency-wise+Channel-wise |
| Accuracy                            | 69.18        | 69.54          | 69.55          | 69.58                       |

Reviewer comment (Reproducibility):
We need an architecture table to compare one (TFC) with the others regarding layer-wise output size and number of parameters.

Author response:
**Q3**: We need an architecture table to compare one (TFC) with the others regarding layer-wise output size and number of parameters.

**A3**: Applying our method or a fast convolution algorithm does not increase the number of parameters in the model or change the size of the layer output. The parameters of the transformation matrix in our algorithms are often incorporated as fixed values in the program or designed for dedicated domain accelerators, rather than being stored as additional parameters. When combining our method with quantization, the storage of the quantized scaling factor is required, but this demand is minimal compared to the number of original parameters in the network.

Reviewer comment (Writing):
The use of “Ternary” in the title is misleading, as ternary refers to an intermediate transform matrix; the actual precision is INT8.

Author response:
**A1**: We apologize for the confuse in our initial preparation. We have updated the name of our proposed algorithm to **Symbolic Fourier Convolution (SFC)**, replacing the previous designation of Ternary Fourier Convolution (TFC). The new name more accurately reflects the underlying concept of our algorithm. The earlier name could potentially be misleading, suggesting the use of three-valued computation throughout the entire convolution process.

Reviewer comment (Evaluation):
One of the primary concerns is the speedup achieved through the current approach. The paper only provides theoretical multiplication complexity reduction, which may not necessarily translate to real-world speedup. Given the complexity of the algorithm and the potential overhead, it is important to provide empirical measurements of computation efficiency.

Author response:
**A2**: Thanks you for your suggestion. **First, we implement our proposed symbolic Fourier convolution algorithm on our Laptop GPU to validate that its can provides real-world speedups**, substantiating not only theoretical computational reductions. For comparison, we evaluated our method against the CUDA Deep Neural Network library (cuDNN) in the context of the 3×3 convolution layer for inference. Because of the time limitation, we were unable to implement the algorithm across entire networks in Int8 format.  As we can see, our algorithms demonstrates a 1.75x speedup compared to cuDNN.

**Table R1. Inference fp32 Conv with BatchSize=1, KSize=3×3，FMSize=28×28，C=128×128 on Nvidia GTX 1650 Laptop GPU**

|      | cuDNN | im2col | SFC  |
| ---- | :---: | ------ | ---- |
| time | 128us | 194us  | 73us |

Furthermore, we opt for **Bit-Operations(BOPs)** as a fine metric to precisely quantify computation costs.  This metric comprehensively considers factors such as **bit-width,  operations number, and the varying hardware costs of addition and multiplication**. And it is widely used in various model compressing fields, including  Neural Architecture Search(NAS)[6], quantization[5] [7] and pruning [5] research.  BOPs for integer addition are defined as $BOPs = b \cdot ADDs$, where '$b$' represents the addition bit-width. Similarly, BOPs for integer multiply-and-accumulate are defined as $BOPs = b1 \cdot b2 \cdot MACs$, where '$b1$' and '$b2$' represent the bit-widths of the two operators, respectively.

For element-wise multiplication, our method consumes $MACs=Tnums\cdot TransSize\cdot TransSize\cdot IC\cdot OC$ when deploying a dense convolution layer, where $ Tnums = ceil(H/TileSize)\cdot ceil(W/TileSize)$. And the input transformation consumes $ADDs = Tnum\cdot ((TileSize+KernelSize)\cdot TransSize)(TileSize+KernelSize+TransSize)\cdot IC$,  the output transformations consumes $ADDs = Tnum\cdot (TileSize\cdot TransSize)(TileSize+TransSize)\cdot OC$.

When the arithmetic bit-width=8, we can find that $BOPs_{ElementWiseMult} >> BOS_{OutTrans} \approx BOS_{InTrans}$, which is why many works based on fast convolution primarily considers only element-wise multiplication numbers when calculating the speedup ratio.

The BOPs of different methods are shown in Table R2. Compared to the state-of-the-art PTQ method AdaQuant[1], SFC method can **significantly reduce the MACs while maintaining narrow bit-width in multiplication**, thus obtaining lower BOPs. In comparison with Winograd-based work[2], our algorithm **does not need to expand the intermediate arithmetic bit-width to maintain the model accuracy**, and thus the BOPs are also lower.

**Table R2. BOPs of ResNet18(69.76%)**

|           | AdaQuant(PTQ)[1] |       | Wino(4×4, 3×3)+QAT[2] | SFC6(7×7, 3×3)+PTQ | SFC4(4×4, 3×3)+PTQ | SFC4(4, 3)+PTQ |
| --------- | ---------------- | ----- | --------------------- | ------------------ | ------------------ | -------------- |
| Bit-width | 8                | 4     | 8/10                  | 8                  | 8/6                | 8/4            |
| Accuracy  | 69.73            | 63.47 | 69.41                 | 69.58              | 69.04              | 63.02          |
| BOPs      | 96.2G            | 24.1G | 44.3G                 | **34.3G**          | **22.8G**          | **15.8G**      |

Reviewer comment (Novelty):
PTQ method is a well-established technique.

Author response:
**A3**: While the PTQ method is a well-established technique, previous work could not perfectly combine it with fast convolutional algorithms to achieve further speedup. For example, the method in ref [3] results in a 2.2%  accuracy reduction for Resnet18 under Int8 PTQ, and in ref [4], there was a 4.6%  accuracy reduction in Resnet18 under Int6 PTQ.  In contrast, our method can be effectively combined with the PTQ with only 0.1% accuracy loss under Int8 and 0.7% under Int6 (Table R2).

Reviewer comment (Experiments):
Today, numerous techniques exist for accelerating CNN models, including the design of compact architecture, sparsity, distillation, and quantization. For quantization alone, it is possible to reduce precision to sub-4 bit level while preserving model accuracy. The paper only compares its approach to other fast convolution algorithms, such as Winograde, and while it demonstrates improvement, it is challenging to evaluate the significance of this approach compared to other alternatives.

Author response:
**A5:** **The motivation of this paper is to enhance the compatibility of fast convolution algorithms with quantization methods, aiming to achieve improved compression performance by combining them**. Fast convolutional algorithms have many advantages, including their **suitability across various hardware platforms**, including CPUs[8], GPUs[9], domain specific accelerators[2]. Other acceleration methods like extreme low-bit quantization and sparsity are more dependent on the support of the hardware architecture for their respective function. For example, Nvidia Turing GPUs can only support specific quantization formats (Int8 and Int16) and sparsity templates (4:2 sparsity).

Discussion in terms of quantization alone: Although achieving sub-4bit quantization and maintaining model accuracy is possible through Quantization-aware Training (QAT) [10], it requires the entire dataset and significant training resources.  In our work, we leverage Post-training quantization (PTQ), which is more lightweight, requiring only a small dataset and estimating parameters in a few minutes. Compared to the sota PTQ alone method AdaQuant[1], our method can achieve **much higher accuracy under similar BOPs or much lower BOPs under similar accuracy**(Table R4). Finally, we believe that our  SFC algorithm can also achieve sub 4-bit quantization via QAT, although we have not attempted it yet.

 **Table R4. BOPs of ResNet18(69.76%)**

|           | AdaQuant(PTQ)[1] | SFC4(4×4, 3×3)+PTQ | SFC4(4, 3)+PTQ |
| --------- | ---------------- | ------------------ | -------------- |
| Bit-width | 4                | 8/6                | 8/4            |
| Accuracy  | 63.47            | **69.04**          | 63.02          |
| BOPs      | 24.1G            | 22.8G              | **15.8G**      |



Discussion in terms of sparsity: The irregular data flow resulting from sparsity poses a significant challenge to hardware architecture design, often leading to hardware inefficiency. Unlike PTQ methods, sparse methods, to maintain accuracy, have high training costs. Furthermore, sparse methods are not incompatible with fast convolutional algorithms; **a previous study [11] explores sparsity in the Winograd domain**. Therefore, we believe that our SFC algorithm can be effectively combined with sparsity methods in the transformation domain.

Discussion in terms of distillation: We think our algorithm is not in conflict with knowledge distillation. Our method provide a fast algorithm for convolution and it is also friendly for quantization. And knowledge distillation is to train a smaller, student model to reproduce the behavior of a larger, teacher model. We can view them as distinct  stages in the model compression pipeline.

Reviewer comment (Reproducibility):
How will this method be used for filters with smaller sizes, such as 1x1 and depth-wise conv layers?

Author response:
**A6**: Our method can accelerate 3×3 and lager size convolutions very efficiently (the only fast convolution algorithm to our knowledge that can accelerate 3×3 convolutions by more than 3× at Int8 without loss of accuracy), but it does not work for 1×1 convolutions. However, our algorithm may affect the application value of the 1×1 convolution. In many compact architectures, 1×1 convolutions are designed to reduce calculations, but can not extract features in the spatial dimension like 3×3. When the overhead of 3×3 convolution is greatly reduced (our method can reduce it by 3.68×), the significance of this design purpose may not be as significant as before. 

Our method can work for depth-wise convolution. However, according to the roof-line model, the inference bottleneck for depth-wise convolution is not in computation but in data transmission(low activation reuse on channel dimension). Thus reducing its computational complexity may not increase its inference speed. Data compression techniques for activation and weights may be more significant for accelerating depth-wise convolution.

Reviewer comment (Writing):
Typo? Page 2, contribution 1 paragraph, x, x, and 7x7.

Author response:
**A8**: Thank you tor your careful reading. We apologize for our oversights. We have corrected these typos in our revised manuscript .

Reviewer comment (Writing):
The writing of this paper is very hard to follow. Many typos are in the paper, such as some numbers are missing in the first contribution of section Introduction " x x ", and the bottom line is missing in Table 2.

Author response:
**Q1**: The writing of this paper is very hard to follow. Many typos are in the paper, such as some numbers are missing in the first contribution of section Introduction " x x ", and the bottom line is missing in Table 2.

**A1**: Thank you tor your careful reading of our paper. We have corrected these typos in our latest manuscript .

Reviewer comment (Novelty):
The motivation for this paper is not clear. Since many compression works like extreme low-bit quantization, pruning, and low-rank decomposition are proposed to accelerate the convolution layers, the motivation using Fourier transformation is not clear.

Author response:
**Q2**: The motivation for this paper is not clear. Since many compression works like extreme low-bit quantization, pruning, and low-rank decomposition are proposed to accelerate the convolution layers, the motivation using Fourier transformation is not clear.

**A2**: The motivation of this paper is to **enhance the compatibility of fast convolution algorithms with quantization methods, aiming to achieve improved computation efficiency by combining them**. Thanks to your suggestion, we have strengthened the expression of motivation in the revised manuscript. Experiment section shows that our method is far ahead of similar work in terms of accuracy, quantization bit-width, and speedup ratio.  

Compared with other compression methods, fast convolution algorithms have many advantages, including their **suitability across various hardware platforms**, including CPUs[8], GPUs[9], domain specific accelerators[2]. Other acceleration methods like extreme low-bit quantization and pruning are more dependent on the support of the hardware architecture for their respective function. For example, Nvidia Turing GPUs can only support specific quantization formats (Int8 and Int16) and pruning templates (4:2 sparsity). 

Discussion in terms of quantization alone: Although achieving sub-4bit quantization and maintaining model accuracy is possible through Quantization-aware Training (QAT) [10], it requires the entire dataset and significant training resources.  In our work, we leverage Post-training quantization (PTQ), which is more lightweight, requiring only a small dataset and estimating parameters in a few minutes. Compared to the sota PTQ alone method AdaQuant[1], our method can **achieve much higher accuracy under similar BOPs** (Bit Operations, which we will detail this metric in **A3** section) or **much lower BOPs under similar accuracy**(Table R1). Finally, we believe that our SFC algorithm can also achieve sub 4-bit quantization via QAT, although we have not attempted it yet.

 **Table R1. BOPs of ResNet18(69.76%)**

|           | AdaQuant(PTQ)[1] | SFC4(4×4, 3×3)+PTQ | SFC4(4, 3)+PTQ |
| --------- | ---------------- | ------------------ | -------------- |
| Bit-width | 4                | 8/6                | 8/4            |
| Accuracy  | 63.47            | **69.04**          | 63.02          |
| BOPs      | 24.1G            | 22.8G              | **15.8G**      |



Discussion in terms of pruning: The irregular data flow resulting from pruning poses a significant challenge to hardware architecture design, often leading to hardware inefficiency. Unlike PTQ methods, pruning methods, to maintain accuracy, have high training costs. Furthermore, pruning methods are not incompatible with fast convolutional algorithms; **a previous study [4] explores sparsity in the Winograd domain**. Therefore, we believe that our SFC algorithm can be effectively combined with pruning methods in the transformation domain.

Discussion in terms of low-rank decomposition: Compared to low-rank decomposition, which provides only **approximate computations**, fast convolution algorithms provide **mathematically equivalent computations**, once numerical errors are within model tolerance, the loss of accuracy is less.

Reviewer comment (Evaluation):
The evaluation is not sufficient. The metric in evaluation is multiplicative complexity, however, the compression ratio and real-time acceleration performance are missing.

Author response:
**Q3**：The evaluation is not sufficient. The metric in evaluation is multiplicative complexity, however, the compression ratio and real-time acceleration performance are missing.

**A3**：Thanks you for your suggestion, we opt for **Bit-Operations(BOPs)** in our revised manuscript as a fine metric to precisely quantify computation costs.  This metric comprehensively considers factors such as **bit-width, the number of operations, and the varying hardware costs of addition and multiplication**. And it is widely used in various model compressing fields, including  Neural Architecture Search(NAS)[6], quantization[5] [7] and pruning [5] research.  BOPs for integer addition are defined as $BOPs = b \cdot ADDs$, where '$b$' represents the addition bit-width. Similarly, BOPs for integer multiply-and-accumulate are defined as $BOPs = b1 \cdot b2 \cdot MACs$, where '$b1$' and '$b2$' represent the bit-widths of the two operators, respectively.

For element-wise multiplication, our method consumes $MACs=Tnums\cdot TransSize\cdot TransSize\cdot IC\cdot OC$ when deploying a dense convolution layer, where $ Tnums = ceil(H/TileSize)\cdot ceil(W/TileSize)$. And the input transformation consumes $ADDs = Tnum\cdot ((TileSize+KernelSize)\cdot TransSize)(TileSize+KernelSize+TransSize)\cdot IC$,  the output transformations consumes $ADDs = Tnum\cdot (TileSize\cdot TransSize)(TileSize+TransSize)\cdot OC$.

When the arithmetic bit-width=8, we can find that $BOPs_{ElementWiseMult} >> BOS_{OutTrans} \approx BOS_{InTrans}$, **which is why many works based on fast convolution primarily considers element-wise multiplication complexity when calculating the speedup ratio.**

The BOPs of different methods are shown in Table R2. Compared to the state-of-the-art PTQ method AdaQuant[1], our SFC method can **significantly reduce the MACs while maintaining narrow bit-width in multiplication**, thus obtaining lower BOPs. In comparison with Winograd-based work[2], our algorithm does **not need to expand the intermediate arithmetic bit-width to maintain the model accuracy**, and thus the BOPs are also lower.

**Table R2. BOPs of ResNet18(69.76%)**

|           | AdaQuant(PTQ)[1] |       | Wino(4×4, 3×3)+QAT[2] | SFC6(7×7, 3×3)+PTQ | SFC4(4×4, 3×3)+PTQ | SFC4(4, 3)+PTQ |
| --------- | ---------------- | ----- | --------------------- | ------------------ | ------------------ | -------------- |
| Bit-width | 8                | 4     | 8/10                  | 8                  | 8/6                | 8/4            |
| Accuracy  | 69.73            | 63.47 | 69.41                 | 69.58              | 69.04              | 63.02          |
| BOPs      | 96.2G            | 24.1G | 44.3G                 | **34.3G**          | **22.8G**          | **15.8G**      |

Reviewer comment (Evaluation):
It would be better to provide the real-time latency to show the acceleration performance on hardware platforms like GPU/FPGA/CPU. The theoretical reduction in the multiplication operation.

Author response:
**Q4**. It would be better to provide the real-time latency to show the acceleration performance on hardware platforms like GPU/FPGA/CPU. The theoretical reduction in the multiplication operation.

**A4**：There has been considerable works on deploying & optimizing Winograd algorithm on CPUs[8] and GPUs[9]. As our algorithm can be computed in the same computational flow as the Winograd, we think its deployment has no serious technical problems.

We implement our proposed symbolic Fourier convolution algorithm on our laptop GPU to validate that its can provides real-world speedups, substantiating not only theoretical computational reductions. For comparison, we evaluated our method against the CUDA Deep Neural Network library (cuDNN) in the context of the 3×3 convolution layer for inference. Because of the time limitation, we were unable to implement the entire networks in Int8 format. As we can see, **our algorithms demonstrates a 1.75x speedup compared to cuDNN**.

**Table R3. Inference Convlayer with BatchSize=1, KernelSize=3×3，FeatureMap=28×28，Channel=128×128 and fp32 format on Nvidia GTX 1650 Laptop GPU**

|      | cuDNN | im2col | SFC4(44, 33)  |
| ---- | :---: | ------ | ---- |
| time | 128us | 194us  | **73us** |

We are also working on FPGA-based CNN accelerator utilizing our proposed algorithm. Welcome to follow our hardware work in the future.

Reviewer comment (Experiments):
Can this method be used in other layers? For example, FC can be seen as a special CNN layer with a kernel size being 1, and what if the proposed method is applied to FC layers?

Author response:
**Q5**: Can this method be used in other layers? For example, FC can be seen as a special CNN layer with a kernel size being 1, and what if the proposed method is applied to FC layers?

**A5**: Our method can accelerate 3×3 and lager size convolutions very efficiently (the only fast convolution algorithm to our knowledge that can accelerate 3×3 convolutions by more than 3× at 8bit with negligible accuracy loss), but it does not work for FC layers. However, in a significant number of CNNs, 3×3 convolution takes up a heavy computational load, much more than FC.