Paper: Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFAC for Large Neural Nets (ICLR-2024)

Reviewer comment (Experiments):
The empirical evaluation is a bit weak. The visualization makes it hard to see how the proposed methods compared to Adam near the end. Also, I think the evaluation will be more reasonable under settings where KFAC outperforms Adam. Such examples can be found in some previous work like [1].

Author response:
> The empirical evaluation is a bit weak. The visualization makes it hard to see how the proposed methods compared to Adam near the end.  The evaluation will be more reasonable under settings where KFAC outperforms Adam.

Our method can indeed outperform AdamW in many models such as CNNs, GNNs and even transformers. We include additional results in Appendix F.
Please also see the general response (2).

Reviewer comment (Reproducibility):
The hierarchical structure seems to be interesting and outperforms the block diagonal structure, but the exact setting for the experiment is a bit unclear. For clarity, it is better to show how model weights with more than two dimensions are merged into two dimensions and the resulting dimensionality.

Author response:
>   why for three of the plots in Figure 9, SINGD uses the diagonal structure instead of the block-diagonal or the hierarchical structure

Note that both block-diagonal and the hierarchical structures include the diagonal structure as a special case.
In these three models, our experiments show that the diagonal structure is good enough to obtain competitive performance.
Thus, we can further reduce the peak memory by using the diagonal structure.
We will add the block-diagonal and the hierarchical structures in the next revision of the paper. 
We will update the caption to clarify this point.

 
In summary, we show that the hierarchical structure can achieve similar performance as the dense structure (INGD) in all the eight models. In some models, the diagonal structure is good enough to match the performance of INGD.  Our approach allows users to choose a range of sparse structures in $\mathbf{K}$ and $\mathbf{C}$ at each layer.

Thank again for asking this follow-up question. Please let us know if you have any questions.

Reviewer comment (Experiments):
I feel like evaluations on settings where KFAC outperforms Adam can convince me more on the effectiveness of the proposed method.

Author response:
> The empirical evaluation is a bit weak. The visualization makes it hard to see how the proposed methods compared to Adam near the end.  The evaluation will be more reasonable under settings where KFAC outperforms Adam.

Our method can indeed outperform AdamW in many models such as CNNs, GNNs and even transformers. We include additional results in Appendix F.
Please also see the general response (2).

Reviewer comment (Writing):
It would have been great to have some more details on the hierarchical approach. Even with the SM, I could not understand it well.

Author response:
We have updated Sec 3.2 to include more thorough discussion on sparse Kronecker factors including the hierarchical structure. Moreover, we further improve Figure 5 (now Figure 6 in Appendix E) and readers can clearly see that the  hierarchical structure give a better approximation while keeping the computational cost low. 
Please also see the general response (1).

Reviewer comment (Evaluation):
From the figures alone, it seems that these methods do no outperform ADAM in terms of convergence to a certain test-loss. What would be the advantage of using this method over ADAM?

Author response:
Our method, using the right settings, can indeed outperform AdamW in many models such as CNNs, GNNs, and even transformers. We include additional results in appendix F. Please also see the general response (2).

Reviewer comment (Writing):
How exactly does the hierarchical approach work?

Author response:
We have updated Sec 3.2 to include more thorough discussion on sparse Kronecker factors including the hierarchical structure. Moreover, we further improve Figure 5 (now Figure 6 in Appendix E) and readers can clearly see that the  hierarchical structure give a better approximation while keeping the computational cost low. 
Please also see the general response (1).