Paper: Coloring Deep CNN Layers with Activation Hue Loss (ICLR-2024)

Reviewer comment (Experiments):
The paper proposes an improvement to the current standard of approaching CNN and superior performance is critical in arguing the value of the idea. While results from table 1 show significant improvement, they are arguable. For instance, in this paper (https://arxiv.org/pdf/2110.00476v1.pdf), there are reports, using the same architectures, of significantly better results. Given that baseline better results are achievable with known techniques, the improvement is shadowed by the probability that the overall technique used in training CNN is flawed and the proposed idea recovers only some of the flaws.

Author response:
Weakness 1, Question 2: The reviewer’s primary concern is that better baseline classification results are available in the literature, and thus any improvement is shadowed by the probability that the overall technique used in training CNN is flawed and the proposed idea recover only some of the flaws.

We agree with the reviewer that baseline accuracy is important, we commit to adding the following information regarding baseline accuracy specifically for Imagenet: Our baseline Resnet18 accuracy 0.635%, 9% lower than reported Resnet18 72.1%[He2016] but is higher than Alexnet 62.5%[Krizhevsky2012]. Our baseline Efficientnet accuracy of 71.4% is closer the reported Efficientnet[Tan2019] 76.3% which uses more elaborate augmentation including random crop

The reviewer is correct, for all datasets used, the highest baseline accuracies are achievable using different models (notably transformers), and higher baseline accuracy for the same models is achievable using extensive data augmentation, e.g. as in the original papers. However, our experiments were constrained by our computational budget for training CNN models, particularly ImageNet training which required on the order of weeks.
our focus was not to achieve the highest possible baseline for specific networks and tasks, which requires extensive augmentation and computationally expensive training

Our goal was not to achieve highest baseline accuracy, but to demonstrate activation hue in most diverse range of contexts. We thus experimented with familiar CNNs and training from scratch, in the context of minimal data augmentation (horizontal flip, no random crop) where convergence may be achieved within 100 epochs. Results are lower that what is achievable in published baselines achieved with more extensive augmentation on individual datasets. For example, On Imagenet, our baseline Resnet18 accuracy 0.635%, 9% lower than reported Resnet18 72.1%[He2016] but is higher than Alexnet 62.5%[Krysevsky2012]. Our baseline Efficientnet accuracy of 71.4% is closer the reported baseline 76.3%[Tan2019] which uses more elaborate augmentation including random crop

Reviewer comment (Experiments):
While there might be major differences between this paper and the referenced paper, making them not directly comparable, this paper must refer to outside sources while establishing baselines. The databases are standard and so are the architectures, therefore other papers should have reported the accuracy for some, if not all, of the cases.

Author response:
We thank the reviewer for raising important points, requesting clarification of the method, and the importance of baseline performance. The primary contention of the review is that because maximum baseline performance is not reported, our results may be flawed and should be rejected. We understand baseline performance is important, however the goal of this paper present the idea of activation hue, linking the hue angle in human color perception to activation hue in deep network machine prediction. Unfortunately due to our limited computational budget, achieving maximum baseline performance (e.g. using extensive data augmentation, hyperparameter optimization) was simply not the best use of resources for this work. We thus opted for a convincing scenario with diverse datasets and tasks, but experiments in the context of limited data augmentation.

We thus hope the reviewer finds our paper acceptable based on the idea of activation hue and limited but convincing experiments. The reviewer may be unfamiliar with color hue, color hue is profoundly connected to human perception and cognition. To our knowledge, this is the first work explore hue angle in multi-channel activation maps, which has possible broad application as other reviewers mention.

Reviewer comment (Writing):
Presentation is not clear and there are problems understanding the method. A critical aspect is that the proposed loss is not explained clearly. For example, on Page 8: "We thus propose a novel loss function leveraging activation hue, that may be used as a training signal in the (x, y) image plane of arbitrary CNN layers, most notably bottleneck layers with minimal spatial extent." There is emphasis on “arbitrary.” Additionally, equations (4) and (5) are written for the final layer only.

Author response:
Weakness 2, Question 1: Regarding the method reviewer states “The idea of regularizing activation maps based on nearest neighbor information is interesting”, however then mentions “I have problems understanding the method”, “the proposed loss is not explained clearly”, “Please explain better how the proposed loss is implemented ”. It appears the reviewer misunderstood the hue loss in sections 4) Method and 5) Experiments, which involves estimating angular labels, and does not involve nearest neighbors as in section 3) Observations. We feel that adding the following text will clarify the implementation:

“We propose a lightweight hue loss term to regularize standard one-hot loss, requiring only two additional network outputs to predict hue-like angular $\theta$ labels parameterized as $(x,y)$ coordinates.”

Reviewer comment (Evaluation):
The paper might benefit from a revision due to a large discrepancy in results, e.g., table 1 Resnet 18 - Cars shows a huge difference between results.

Author response:
Regarding larger differences in Table 1, e.g. “Resnet18 - Cars”, Resnet18 had simply converged to suboptimal solutions due to limited augmentation, also Efficientnet several cases of fine-grained classification. This highlights the possibility of activation hue regularization where limited augmentation is possible.

Reviewer comment (Reproducibility):
Please explain better how the proposed loss is implemented.

Author response:
Weakness 2, Question 1: Regarding the method reviewer states “The idea of regularizing activation maps based on nearest neighbor information is interesting”, however then mentions “I have problems understanding the method”, “the proposed loss is not explained clearly”, “Please explain better how the proposed loss is implemented ”. It appears the reviewer misunderstood the hue loss in sections 4) Method and 5) Experiments, which involves estimating angular labels, and does not involve nearest neighbors as in section 3) Observations. We feel that adding the following text will clarify the implementation:

“We propose a lightweight hue loss term to regularize standard one-hot loss, requiring only two additional network outputs to predict hue-like angular $\theta$ labels parameterized as $(x,y)$ coordinates.”

Reviewer comment (Experiments):
The critical question is about comparison: is the mentioned reference relevant? If not, can we find another paper establishing baselines for the tests?

Author response:
We thank the reviewer for raising important points, requesting clarification of the method, and the importance of baseline performance. The primary contention of the review is that because maximum baseline performance is not reported, our results may be flawed and should be rejected. We understand baseline performance is important, however the goal of this paper present the idea of activation hue, linking the hue angle in human color perception to activation hue in deep network machine prediction. Unfortunately due to our limited computational budget, achieving maximum baseline performance (e.g. using extensive data augmentation, hyperparameter optimization) was simply not the best use of resources for this work. We thus opted for a convincing scenario with diverse datasets and tasks, but experiments in the context of limited data augmentation.

We thus hope the reviewer finds our paper acceptable based on the idea of activation hue and limited but convincing experiments. The reviewer may be unfamiliar with color hue, color hue is profoundly connected to human perception and cognition. To our knowledge, this is the first work explore hue angle in multi-channel activation maps, which has possible broad application as other reviewers mention.

Weakness 1, Question 2: The reviewer’s primary concern is that better baseline classification results are available in the literature, and thus any improvement is shadowed by the probability that the overall technique used in training CNN is flawed and the proposed idea recover only some of the flaws.

We agree with the reviewer that baseline accuracy is important, we commit to adding the following information regarding baseline accuracy specifically for Imagenet: Our baseline Resnet18 accuracy 0.635%, 9% lower than reported Resnet18 72.1%[He2016] but is higher than Alexnet 62.5%[Krizhevsky2012]. Our baseline Efficientnet accuracy of 71.4% is closer the reported Efficientnet[Tan2019] 76.3% which uses more elaborate augmentation including random crop

The reviewer is correct, for all datasets used, the highest baseline accuracies are achievable using different models (notably transformers), and higher baseline accuracy for the same models is achievable using extensive data augmentation, e.g. as in the original papers. However, our experiments were constrained by our computational budget for training CNN models, particularly ImageNet training which required on the order of weeks.
our focus was not to achieve the highest possible baseline for specific networks and tasks, which requires extensive augmentation and computationally expensive training

Reviewer comment (Experiments):
It would be better to focus on a set of architectures and use them over the entire paper. Currently, some models are used in explanations, others in evaluation, and the paper is not convincing. For example, the sentence "Similar improvements were observed with a variety of different network architectures including VGG, Inception v3, DenseNet" suggests that results could have been shown as additional material.

Author response:
Regarding results for Table 1, not including VGG, Inception, Densenet. While various networks were important to motivate observations hue in Section 3), they were all outperformed by Efficientnet, particularly on smaller datasets. We thus presented the most improved (Resnet18) and best result (EfficientNetB0), to avoid redundancy.

Reviewer comment (Experiments):
The paper does not provide evaluation results by properly scaling the employed models, e.g., applying the approach to ResNet-50 or a larger one. Thus, it is difficult to assess the extent of improvement brought by the proposed method.

Author response:
Weaknesses 1,3: The comment regarding scaling to larger networks is well taken, we propose to mentioning this in the discussion of future work applying activation hue to larger networks, tasks with spatial extend including detection, segmentation, and alternative architectures such as Vision Transformer mentioned by another reviewer. Activation hue may in principle be

Reviewer comment (Experiments):
The proposed activation hue's properties should be discussed along with experiments. Will it improve the CNN network converge, or make it robust to some perturbations?

Author response:
Weaknesses 2, 4, Question 1: Regarding implementing activation hue, complexity, we add emphasize the lightweight nature of the implementation throughout the paper (abstract, introduction, Section 4) Method, Experiments. We will also provide a github code link in footnotes, which we neglected in the original submission. The following sentence is added:

“We propose a lightweight hue loss term to regularize standard one-hot loss, requiring only two additional network outputs to predict hue-like angular $\theta$ labels from $(x,y)$ coordinates.”

Weaknesses 2: regarding properties, convergence, functioning of the method: activation hue loss is implemented by a pair of output neurons which constrain network weights to be predictive of angular (x,y) class labels. We hypothesize this encourages and captures an activation hue intrinsic to each class, which leads to improved convergence.

Reviewer comment (Experiments):
More results about employing the given method to downstream tasks, e.g., detection, and segmentation, will further validate its effectiveness and generality.

Author response:
Weaknesses 1,3: The comment regarding scaling to larger networks is well taken, we propose to mentioning this in the discussion of future work applying activation hue to larger networks, tasks with spatial extend including detection, segmentation, and alternative architectures such as Vision Transformer mentioned by another reviewer. Activation hue may in principle be

Reviewer comment (Reproducibility):
The complexity of the introduction of the novel hue-like parameter to the model architecture and training process should be discussed.

Author response:
Weaknesses 2, 4, Question 1: Regarding implementing activation hue, complexity, we add emphasize the lightweight nature of the implementation throughout the paper (abstract, introduction, Section 4) Method, Experiments. We will also provide a github code link in footnotes, which we neglected in the original submission. The following sentence is added:

“We propose a lightweight hue loss term to regularize standard one-hot loss, requiring only two additional network outputs to predict hue-like angular $\theta$ labels from $(x,y)$ coordinates.”

Reviewer comment (Presentation):
There may exist some misuse between \cite and \citep as some citation formats seem improper in the paper.

Author response:
Fixed possible citep

Reviewer comment (Evaluation):
Table 1 shows that the given method yields better performance in fine-grained classification than that in common ones. Any further explanations for them?

Author response:
Question 2: Fine grained results: it is true improvements are more marked for certain fine-grained tasks, Birds for Efficientnet, and Cars for Resnet18. It appears that Resnet may have difficulty converging given minimal augmentation in our experimental protocol (horizontal flips), may be related to the distinctiveness of the activation hue for these classes. We would emphasize this in a Discussion sentence.

Reviewer comment (Experiments):
The experiments are conducted with traditional architectures. How about applying the proposed method for training more advanced architecture, such as Vision Transformer?

Author response:
We agree the activation hue principle could be broadly applied to other architectures, tasks. We propose add a Discussion sentence to mention potential application to the Vision Transformer, in additional to detection and segmentation tasks, scaling to larger models as mentioned by other reviewers.

Reviewer comment (Experiments):
There are already some well-known techniques based on the similarity of activation vectors, such as label smooth. I think the related methods should be compared or discussed.

Author response:
We thank the reviewer for mentioning label smoothing, we add this to our related work as follows, it is similar to the work of Rodriguez 2018 ‘beyond one-hot encoding’ in seeking to reformat the shape of training signals.

“Our final results training from scratch using one-hot $+$ hue loss are similar in spirit to work seeking to regularize the label and/or the activation space, including using real-valued rather than strictly one-hot training labels \cite{rodriguez2018beyond}, label smoothing which adds a constant to one-hot labels\cite{muller2019does},”

Reviewer comment (Evaluation):
The extra regularization term seems to introduce extra computation costs. I think the comparison of computation should be presented.

Author response:
Similar comments from other reviewers allow us to recognize we did not sufficiently describe our implementation. The computational cost of implementing activation hue is minimal, equivalent to two extra output neurons for angular labels in x, y coordinate format.

"We propose a lightweight hue loss term to regularize standard one-hot loss, requiring only two additional network outputs to predict hue-like angular $\theta$ labels parameterized as $(x,y)$ coordinates."