Paper: NeuralFuse: Learning to Recover the Accuracy of Access-Limited Neural Network Inference in Low-Voltage Regimes (ICLR-2024)

Reviewer comment (Evaluation):
The transferability on different error rate and model size is not very good, according to Table 1, which means re-training is still required for different model/dataset/SRAM voltages.

Author response:
As shown in Table 1, the performance of transferability would be better when the source model and the target model are the same or similar.  Besides, in Appendix H.4, we also report the ensemble-based results, which means that we use two surrogate models to train NeuralFuse. The experimental results show that training with ensemble surrogate models can receive better transferability than only using one mentioned in Table 1. Please note that this is also a trade-off, and one should choose similar models to transfer the NeuralFuse in a limited access scenario as much as possible.

Reviewer comment (Evaluation):
The energy saving is only ~20% by reducing SRAM voltage, while the accuracy drop is beyond 1%. It needs some justification on this trade-off.

Author response:
We would like to point out that the accuracy drop shown in our paper was an experimental scenario to demonstrate the efficacy of NeuralFuse. Of course, one would not want to suffer a significant drop in accuracy, but it is intrinsically a trade-off between accuracy drop and energy saving. That being said, in practice, NeuralFuse can be used to adjust the voltage level (corresponding to different BERs) to achieve the purpose of energy saving, while having acceptable accuracy, which also reflects the adaptivity and generality of NeuralFuse.

Reviewer comment (Evaluation):
The net benefits of introducing NeuralFuse in tandem with low-voltage operation remain uncertain. While there are energy savings associated with SRAM accesses, these reports overlook the comprehensive energy consumption of NeuralFuse, particularly the MAC operations.

Author response:
We understand the reviewers' concerns, therefore, we provide the approximation of the end-to-end energy-saving results based on MACs in [Global Response](https://openreview.net/forum?id=Qvoe4wXWFi&noteId=X3C19r9SKy), which can provide more insights into total energy saving under low-voltage scenarios. We observe that only VGG11 with two larger NeuralFuse (ConvL and DeConvL) may increase the total energy. On the contrary, others’ pairs still can save a large amount of energy, which is consistent with the results reported in Table 2.

Reviewer comment (Evaluation):
The significant fluctuation in accuracy suggests that the optimized model may lack consistent predictability.

Author response:
We note that random fluctuation is caused by the random bit flipping caused by low-voltage, and hence larger models, which have more parameters, will suffer more accuracy fluctuation from it. Therefore, the fluctuation is the nature of this challenging task we are addressing (recovering the accuracy against random bit errors), instead of a weakness of our solution.

Reviewer comment (Evaluation):
The true efficacy of power savings from the low-voltage operation remains ambiguous. While Table 2 highlights energy savings, it narrowly focuses on the consumption related to SRAM accesses. Given that the large configurations of NeuralFuse exhibit similar MACs to the base models (as seen in Table 7), the feasibility of NeuralFuse, when accounting for its total overhead, merits reconsideration.

Author response:
We understand the reviewers' concerns, therefore, we provide the approximation of the end-to-end energy-saving results based on MACs in [Global Response](https://openreview.net/forum?id=Qvoe4wXWFi&noteId=X3C19r9SKy), which can provide more insights into total energy saving under low-voltage scenarios. We observe that only VGG11 with two larger NeuralFuse (ConvL and DeConvL) may increase the total energy. On the contrary, others’ pairs still can save a large amount of energy, which is consistent with the results reported in Table 2.

Reviewer comment (Evaluation):
The unpredictability of model performance under low voltage operation, especially with bit flips at the MSBs, poses challenges for practical implementation. Could you shed more light on its real-world applicability or potential use-cases?

Author response:
This is really a well-thought-out comment! Previous works have pointed out some possible scenarios that suffer from energy concerns and hence need some strategies to reduce energy consumption. For example, in [2, 3], the authors mention that due to the high computation cost of CNN processing and some DNN-based vision algorithms, they will incur high energy consumption. This will significantly reduce the battery life of battery-powered devices, indirectly impacting the user experience of the devices. Therefore, to avoid the aforementioned issues, we can mitigate the device's energy consumption by lowering the operating voltage and then incorporating NeuralFuse to recover model performance, reducing the side effects caused by low voltage.

Reviewer comment (Evaluation):
Lack of discussion of introduced overhead of the generator modules. For ImageNet-10, the best generator architecture, UNet-L, has 2.03G MACs. The introduced extra computation cost is significant compared to the vanilla model (ResNet18 only has 1.82 G MACs). It raises the concern that the introduced overhead for the generator is too large compared to the classifier, making the proposed strategy unrealistic. The author only discusses the energy of SRAM access without considering the computation energy and latency.

Author response:
We understand the reviewers' concerns; we acknowledge that when evaluating the ImageNet-10 dataset, utilizing UNetL with base model ResNet18, the resulting energy consumption would be higher than the vanilla model. However, we would like to emphasize that this is a trade-off, and it should not be considered unrealistic. When utilizing NeuralFuse to recover the low-voltage-induced accuracy drop, a suitable architecture of the NeuralFuse generator should be considered to reduce the overall energy consumption. Understandably, one utilizing a small base model should not choose such a NeuralFuse generator that is larger than it.  Take ResNet18 for example, we can choose UNetS, which has the MAC value of only 518.47M (less than ResNet18, 1.82G) and can still recover the accuracy up to 86%.

Regarding the computation energy, we did not report the *true* end-to-end energy consumption as it will depend on various factors (as mentioned in Section 4.1). However, MACs (multiply-accumulate operations) report the computation required to perform a forward pass through a neural network [1, 2, 3, 4], and we believe it is a sufficient representation of the energy consumption of the computation. Therefore, we provide the approximation of the end-to-end energy consumption by using MACs in [Global Response](https://openreview.net/forum?id=Qvoe4wXWFi&noteId=X3C19r9SKy).

In the table below, we report the latency (batch_size=1, CIFAR-10/ImageNet-10 testing dataset) of utilizing the different NeuralFuse generators with two different base models, ResNet18 and VGG19. We can see that although NeuralFuse indeed brings a certain degree of extra latency, we argue that this is an unavoidable factor; however, since the latency is measured on a general-purpose GPU (i.e. V100), when the base model and NeuralFuse are deployed on a custom accelerator, we believe this delay will be further reduced.

[NeuralFuse Extra Latency Compared to Only Base Model] 
| | ResNet18 (CIFAR-10) | VGG19 (CIFAR-10) | ResNet18 (ImageNet-10) | VGG19 (ImageNet-10) |
|-:|-:|-:|-:|-:|
| Base Model|5.84 ms|5.32 ms|6.21 ms|14.34 ms|
|+ ConvL | 9.37 ms (+3.53)|8.96 ms (+3.64)|10.51 ms (+4.30)|17.66 ms (+3.32)|
|+ ConvS | 7.86 ms (+2.02)|7.40 ms (+2.08)|8.28 ms (+2.07)|16.72 ms (+2.38)|
|+ DeConvL | 9.18 ms (+3.34)|8.59 ms (+3.27)|10.07 ms (+3.86)|17.24 ms (+2.90)|
|+ DeConvS | 7.49 ms (+1.65)|7.04 ms (+1.72)|7.79 ms (+1.58)|15.67 ms (+1.33)|
|+ UNetL | 10.69 ms (+4.85)|10.06 ms (+4.74)|11.14 ms (+4.93)|18.54 ms (+4.20)|
|+ UNetS | 10.63 ms (+4.79)|10.13 ms (+4.81)|11.36 ms (+5.15)|18.60 ms (+4.26)|

Reviewer comment (Evaluation):
The introduced generator modules may dilute the energy efficiency brought by the low-voltage scheme. Based on Appendix E, the total computations are very large. A more ideal accuracy-saving method should introduce less overhead.

Author response:
We understand the reviewer’s concern. Actually, in Table 2, we have already considered both the base model and NeuralFuse energy consumption to calculate the energy saving ratio. Besides, we also show the energy saving results based on MACs in [Global Response](https://openreview.net/forum?id=Qvoe4wXWFi&noteId=X3C19r9SKy), which consider the overall energy consumption. The results show that only VGG11 with two larger NeuralFuse (ConvL and DeConvL) may increase the total energy. On the contrary, others’ pairs still can save a large amount of energy, consistent with our original findings. Although our approach dilutes the amount of energy saving achieved from low voltages, given that our approach is model-agnostic and more broadly applicable to a variety of scenarios, especially models with limited access. This also demonstrates the versatility of NeuralFuse, including adaptability, portability, transferability, and generality.

Reviewer comment (Evaluation):
Could the author provide a more complete overhead analysis of the introduced generator? The author should show the introduced energy cost of computation (both memory and computation) and extra latency overhead in 4.4. The paper would be more meaningful if it saved accuracy under small overhead.

Author response:
We understand the reviewers' concerns; we acknowledge that when evaluating the ImageNet-10 dataset, utilizing UNetL with base model ResNet18, the resulting energy consumption would be higher than the vanilla model. However, we would like to emphasize that this is a trade-off, and it should not be considered unrealistic. When utilizing NeuralFuse to recover the low-voltage-induced accuracy drop, a suitable architecture of the NeuralFuse generator should be considered to reduce the overall energy consumption. Understandably, one utilizing a small base model should not choose such a NeuralFuse generator that is larger than it.  Take ResNet18 for example, we can choose UNetS, which has the MAC value of only 518.47M (less than ResNet18, 1.82G) and can still recover the accuracy up to 86%.

Regarding the computation energy, we did not report the *true* end-to-end energy consumption as it will depend on various factors (as mentioned in Section 4.1). However, MACs (multiply-accumulate operations) report the computation required to perform a forward pass through a neural network [1, 2, 3, 4], and we believe it is a sufficient representation of the energy consumption of the computation. Therefore, we provide the approximation of the end-to-end energy consumption by using MACs in [Global Response](https://openreview.net/forum?id=Qvoe4wXWFi&noteId=X3C19r9SKy).

In the table below, we report the latency (batch_size=1, CIFAR-10/ImageNet-10 testing dataset) of utilizing the different NeuralFuse generators with two different base models, ResNet18 and VGG19. We can see that although NeuralFuse indeed brings a certain degree of extra latency, we argue that this is an unavoidable factor; however, since the latency is measured on a general-purpose GPU (i.e. V100), when the base model and NeuralFuse are deployed on a custom accelerator, we believe this delay will be further reduced.

[NeuralFuse Extra Latency Compared to Only Base Model] 
| | ResNet18 (CIFAR-10) | VGG19 (CIFAR-10) | ResNet18 (ImageNet-10) | VGG19 (ImageNet-10) |
|-:|-:|-:|-:|-:|
| Base Model|5.84 ms|5.32 ms|6.21 ms|14.34 ms|
|+ ConvL | 9.37 ms (+3.53)|8.96 ms (+3.64)|10.51 ms (+4.30)|17.66 ms (+3.32)|
|+ ConvS | 7.86 ms (+2.02)|7.40 ms (+2.08)|8.28 ms (+2.07)|16.72 ms (+2.38)|
|+ DeConvL | 9.18 ms (+3.34)|8.59 ms (+3.27)|10.07 ms (+3.86)|17.24 ms (+2.90)|
|+ DeConvS | 7.49 ms (+1.65)|7.04 ms (+1.72)|7.79 ms (+1.58)|15.67 ms (+1.33)|
|+ UNetL | 10.69 ms (+4.85)|10.06 ms (+4.74)|11.14 ms (+4.93)|18.54 ms (+4.20)|
|+ UNetS | 10.63 ms (+4.79)|10.13 ms (+4.81)|11.36 ms (+5.15)|18.60 ms (+4.26)|

Reviewer comment (Experiments):
Could the author compare with other error-mitigation methods for bit error in SRAM?

Author response:
We have provided the experimental results based on a robust model trained with adversarial weight perturbation in Appendix M. Our results show that although the weights are perturbed during training, the trained robust models still can not resist the random bit flips (The perturbed accuracy of PerAct ResNet18 is only about 23%). The results also show that using input transformation based on NeuralFuse is indeed for recovering the accuracy.

Reviewer comment (Experiments):
The review can see the architectures of the generators in the appendix. How are the detailed architecture of generators decided? Any insights on the architecture of the NeuralFuse generator?

Author response:
Thank you for raising this fundamental question. When it comes to deciding which architecture should be adopted to address this particular issue, we have two main goals: 1) efficiency (so the overall energy overhead is decreased) and 2) robustness (so that it can generate robust patterns on the input image and overcome the random bit flipping in subsequent models). As mentioned in Section 4.1, the design of ConvL is inspired by Nguyen and Tran [2], in which the authors utilize a similar architecture to design an input-aware trigger generator, and have demonstrated its efficiency and effectiveness. Furthermore, since ConvL is an encoder-decoder-based model, we attempted to enhance it by replacing the *Upsampling* layer with a *Deconvolution* layer, leading to the creation of DeConvL. The UNetL-based NeuralFuse draws inspiration from Ronneberger et al. [3], known for its robust performance in image segmentation, and thus, we incorporated it as one of our architectures. Lastly, ConvS, DeConvS, and UNetS are *scaled-down* versions of the model designed to reduce computational costs and total parameters. These architectural variations also demonstrate the trade-off between energy saving and the accuracy recovering rate, and a practitioner can specify a particular architecture based on the need.