Paper: Fixed Non-negative Orthogonal Classifier: Inducing Zero-mean Neural Collapse with Feature Dimension Separation (ICLR-2024)

Reviewer comment (Theory):
Why do we have to fix classifier as an orthogonal matrix?

Author response:
The reason why fixed classifiers have non-negativity and orthogonality constraints is that if those constraints do not exist in the LPM with a fixed classifier (not a simplex), it is not guaranteed that the collapse phenomenon between class means and their respective class weight vectors occurs when the LPM has the optimal solution.  
However, when the fixed classifier becomes a non-negative and orthogonal matrix, the LPM obtains the optimal solution even in inducing max margin in decision while the collapse occurs.

Reviewer comment (Theory):
Does Remark.1 claim that zero-mean neural collapse can achieve max-margin? Considering the binary class classification, the max-margin feature should be Digon, which has the larger angle (180 degrees) than orthogonality (90 degrees).

Author response:
According to the constraints of Eq. 5 in the revised paper, last-layer features are located on the non-negative Euclidean space because they followed the ReLU activation function and were centered at the origin.  
In this situation, the max-margin in decision becomes orthogonal.

Reviewer comment (Novelty):
The case that D > K is interesting. Authors can refer to [1] https://en.wikipedia.org/wiki/Thomson_problem and [2] https://arxiv.org/abs/2310.05351.

Author response:
Thanks for your thoughtful references about the future work.  
We reflexted them to the future work paragraph in Section 8 (Conclusion and Future Work) on page 9 of the revised paper.  
We don't doubt that these materials will be helpful to resolve the limitation.

Reviewer comment (Theory):
Although the proposed methods are effective, the connections with the theoretical analysis seem unclear.

Author response:
To improve the connections, we changed the order of paragraphs in Section 5 on page 5 of the revised paper according to the modified logical flow of detailed explanation of our motivation in W1.
Additionally, we divided our previous Figure 1 by splitting it into three separate figures (Figure 1, 2, and 3).
Figure 1 now illustrates the main motivation of our work, which is *how the collapse between class means and class weight vectors occurs in fixed classifier?*.
Figure 2 describes the our method, FNO classifier.
Figure 3 provides the utilization of feature dimension separation in continual learning and imbalanced learning.
We highlight the impact of the FNO classifier, which enhances masked softmax for continual learning and adjusts the original mixup on the hypersphere by using *arc-mixup* for imbalanced learning.
Please refer to Figure 1, 2, and 3 on pages 1, 5, and 6 of the revised paper for a clear visualization, respectively.

Reviewer comment (Experiments):
It will be more convincing if more competitors on ImageNet-LT and Places-LT are provided.

Author response:
We borrowed the performance of ETF in ImageNet-LT from [A1].
Additionally, to ensure a fair comparison, we reproduced the ETF classifier with the same hyperparameter settings to ours for CIFAR10/100-LT, ImageNet-LT and Places-LT.
Please refer to Tables 3 and 4 on page 9 and Tables 9, 10, and 11 on page 28 of the revised paper.

[A1] Yibo Yang, Liang Xie, Shixiang Chen, Xiangtai Li, Zhouchen Lin, and Dacheng Tao. Do we really
need a learnable classifier at the end of deep neural network? arXiv preprint arXiv:2203.09081,
2022b.

Reviewer comment (Experiments):
The experiments in the paper are limited to continual and imbalanced learning scenarios for the FNO classifier. It would be beneficial to see how the FNO classifier performs compared to the ETF classifier in standard classification tasks. Additionally, in Table 4, which details the imbalanced learning experiments, the ETF classifier is absent from the comparison. Including it could provide a more comprehensive evaluation of the FNO classifier's performance.

Author response:
**Weakness 1-1.** The experiments in the paper are limited to continual and imbalanced learning scenarios for the FNO classifier. It would be beneficial to see how the FNO classifier performs compared to the ETF classifier in standard classification tasks.

|Architecture|Method|MNIST|Fashion MNIST|CIFAR10|CIFAR100|
|----|----|----|----|----|----|
|ResNet|FC|99.64|93.56|89.45|65.11|
|ResNet|ETF|99.66|93.82|88.05|64.23|
|ResNet|FNO|99.67|94.28|89.59|66.20|
> We conducted experiments of ETF in standard classification tasks.
For a fair comparison, we used the same environmental settings to ours when utilizing the analyses of ZNC in Appendix B.1 (Zero-Mean Neural Collapse in Image Classification Benchmarks) on page 20 of the revised paper.
As shown the above table, our FNO outperforms ETF in standard classification tasks.

Reviewer comment (Experiments):
For a full comparison, we may also want to consider the incorporation of MSE loss within the ETF classifier, which is guaranteed to be an orthogonal classifier (assuming no bias).

Author response:
**Weakness 2-2.** For a full comparison, we may also want to consider the incorporation of MSE loss within the ETF classifier, which is guaranteed to be orthogonal classifier (assuming no bias)
> When incorporating MSE loss within the fixed ETF classifier, the shape where trained class means converge will vary while aligning to their respective class weight vectors in a similar way to [A1], *i.e.*, it is not guaranteed that the class means collapse their respective class weight vectors at the orthogonal matrix when the fixed classifier is not orthogonal.

Reviewer comment (Writing):
The manuscript's writing and structure require refinement for better clarity and flow.

Author response:
To clarify our motivation, we highlight that "the collapse does not occur at a simplex ETF in the fixed classifier when the shape of weight matrix in it is not a simplex ETF".
Additionally, we included more specific visual materials for better comprehension in Figure 1 on page 1 of the revised paper.

Reviewer comment (Writing):
The concepts of masking in continual learning and mixup in imbalanced learning emerge unexpectedly within the text and would benefit from a better introduction with an improved link with neural collapse.

Author response:
To enhance the coherence of our paper's structure, we more specifically introduce the masking concept in continual learning and arc-mixup adjusted on the hypersphere in imbalanced learning:
 "In addition, the FNO classifier becomes linearly independent due to non-negativity and orthogonality.
As a result, some elements in the last-layer feature engaged to one class will not be able to affect other classes.
This *feature dimension separation* (FDS) synergizes with masked softmax by decreasing interferences between classes in continual learning and enables *arc-mixup* by adjusting the mixup strategy on the hypersphere to work correctly in imbalanced learning."

*All modifications were reflected in Section 1 (Introduction) on page 2 of the revised paper.
We anticipate that these additions will facilitate readers' comprehension of our work.*

Reviewer comment (Theory):
The significance of Zero-mean Neural Collapse (ZNC) in non-negative Euclidean space (i.e., the positive hyper-octant) is not immediately apparent. The paper should clarify whether its importance is solely due to the optimality shown in the LPM model or if there are additional factors which are outside the proof. The rationale behind constraining the representation space to the positive hyper-octant warrants further explanation.

Author response:
**(Clarify ZNC's importance in the LPM model.)** The reason why fixed classifiers have non-negativity and orthogonality constraints is that if those constraints do not exist in LPM with a fixed classifier (not a simplex), it is not guaranteed that the collapse phenomenon between class means and their respective class weight vectors occurs when LPM has the optimal solution.
However, when the fixed classifier becomes a non-negative and orthogonal matrix, LPM obtains the optimal solution even in inducing max margin in decision while the collapse occurs.
Then, we have proven the utility of those constraints, and in order to explain the collapse phenomenon that occurs when LPM with a fixed non-negative orthogonal classifier achieves the optimal solution, it becomes necessary to define special neural collapse properties: zero-mean neural collapse.

**(Additional factors.)** Upon the constraints of ZNC, non-negativity and orthogonality, we propose the fixed non-negative orthogonal classifier, figure out their additional benefit (called *feature dimension split*), and utilize our method with masked softmax in continual learning and arc-mixup in imbalanced learning.

Reviewer comment (Theory):
The nature of the problem posed by the LPM model is not shown. The manuscript should specify whether it is linear, non-linear, or solvable by known matrix factorization techniques. Moreover, the discussion on the complexity of providing values for W is insufficiently developed, leaving the reader questioning where the complexity of the problem truly lies.

Author response:
Non-negativity and orthogonality are highly beneficial for LPM with a fixed classifier to have the optimality while inducing the collapse.
As a result, all class weight vectors are restricted on the positive hyper-octant.
In addition, the last-layer features follow the ReLU and are centered at the origin.
Therefore, all weights and features are located on the positive hyper-octant.

*Following the above logical flow, we properly changed the order of paragraphs in Section 5 on page 5 of the revised paper.*

Reviewer comment (Theory):
Could the authors provide insight into why LPM optimality does not manifest in the case of a regular fixed d-simplex, and conversely, why it appears to be present in the context of Zero-mean Neural Collapse (ZNC)?

Author response:
[A1] introduced Layer-Peeled Model (LPM), for Eq. 4 *(on page 3 of our revised paper)* with the goal of unveiling quantitative patterns of deep neural networks.
The last-layer features $H$ comes from the previous $L-1$ layers which have a non-linear activation function such as the ReLU, so LPM in the new optimization program is non-linear.

**(In terms of general LPM model)** [A1] introduced Layer-Peeled Model (LPM), for Eq. 4 *(on page 3 of our revised paper)* with the goal of unveiling quantitative patterns of deep neural networks.
This new optimization program remains non-convex  but is likely much more conductive to analysis than the deep neural networks by considering only last-layer features and classification layer.

**(In terms of our LPM model)** The primary objective of LPM with a fixed classifier (not a simplex) is to observe how the collapse happens when the LPM has an optimal solution.
In this case, it is not assured that the class means and class weight vectors will collapse in a simplex ETF.
As a result, our main challenge in the LPM with a fixed classifier becomes to identify the appropriate constraints for the LPM to have the optimal solution while the collapse occurs, not to optimize the optimal solution of $W$ (because it is fixed).

[A1] Cong Fang, Hangfeng He, Qi Long, and Weijie J. Su. Exploring deep neural networks via layer-peeled-model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences, 118(43), oct 2021.

Reviewer comment (Presentation):
The tables detailing experimental results should more clearly differentiate the methodologies used, to avoid confusion. The complex nomenclature, such as FNODERMR++, could be simplified for better clarity.

Author response:
We have revised our previous Figure 1 by splitting it into three separate figures (Figure 1, 2, and 3).
Figure 1 depicts how the collapse between class means and class weight vectors occurs in fixed classifier.
Figure 2 describes the our method, FNO classifier.
Figure 3 provides an overview of two applications of the FNO classifier.
We highlight the impact of the FNO classifier, which enhances masked softmax for continual learning and adjusts the original mixup on the hypersphere by using *arc-mixup* for imbalanced learning.
Please refer to Figure 1, 2, and 3 on pages 1, 5, and 6 of the revised paper for a clear visualization, respectively.

Reviewer comment (Writing):
In Remark 1 at the end of section 5, the statement regarding the inability of a fixed orthogonal classifier to address neural collapse needs further clarification. A more detailed explanation could help in understanding this assertion.

Author response:
We divided the column of Method in original Table 2 into three parts: RM (indicating rehearsal-based method used), Clf (indicating classifier used), M (indicating whether a negative infinite mask was applied or not).
The abbreviations FC and FNO indicate a learnable classifier and our FNO classifier, respectively.
All modifications were reflected and please refer to Table 2 on page 8 of the revised paper.