Paper: Dictionary Contrastive Learning for Efficient Local Supervision without Auxiliary Networks (ICLR-2024)

Reviewer comment (Writing):
The authors mention the mutual information $I$, but never formally define it. Furthermore, even though the authors give details about how they estimate mutual information, an explicit pointer (e.g., “Appendix D describes how we estimate the mutual information”) in the text could make the paper easier to read.

Author response:
Many thanks for this advice and suggestion. Following Reviewer SF1W’s feedback, we have included the definition of mutual information in Section 4.1. That is, we added the definition “mutual information $I$ signifies the amount of information obtained about one random variable by observing another.” and its mathematical definition to Appendix B, where we use the definition to derive Eq. (13) from Eq. (12). Furthermore, in Section 5.3.1, which focuses on the mutual information experiment, we now include a pointer to Appendix D.2.6 for detailed information on the estimation methods used.

Reviewer comment (Writing):
While in eqn. $i$ and $j$ are used to refer to sample indices, in eqn. 2 the authors use $n = 1, \cdots, N$. Index $i$ in eqn. 2 then means the class $i = 1, \cdots, Z$ and elements of the dictionary are referred to as $\mathbf{t}\_{z} \in \mathbf{D}^{Z}$, which is confusing as well, as $z$ can be easily mistaken for $\mathbf{z} = f_{\phi}(\mathbf{h})$. Furthermore, on page 3, $y \in \mathbb{N}$ would imply an infinite amount of classes. Authors should refer to $y \in \{1, \cdots, Z\}$, as they do in Table 4. The fact that one has a finite number of classes should be explicit.

Author response:
Thank Reviewer SF1W for highlighting the issue with the variable $z$. To reduce confusion, we have revised the notation for auxiliary network output from $\boldsymbol{z}=f_\phi(\boldsymbol{h})$ to $\boldsymbol{a}=f_\phi(\boldsymbol{h})$. Additionally, to ensure consistency, we have unified the class index as 'z' in Eq  (2) and Appendix B, and changed $y\in\mathbb{N}$ to $y\in\{1,...,Z\}$ on page 3.

Reviewer comment (Writing):
The paper contains some reference errors, especially with respect to arXiv references to conference papers. Here is a non-exhaustive list of wrong references:  
- Yulin Wang, Zanlin Ni, Shiji Song, Le Yang, and Gao Huang. Revisiting locally supervised learning: an alternative to end-to-end training. arXiv preprint arXiv:2101.10832, 2021. (Published in ICLR 2021)  
- Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. (Published in ICLR 2015)

Author response:
Thanks for pointing out our mistakes on citations. We have revised the citation.  

[1] Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep
representations. The Journal of Machine Learning Research, 19(1):1947–1980, 2018.


[2] Yulin Wang, Zanlin Ni, Shiji Song, Le Yang, and Gao Huang. Revisiting locally supervised learning:
an alternative to end-to-end training. In International Conference on Learning Representations,
2020

Reviewer comment (Theory):
While the authors show in Appendix B that minimizing $\mathcal{L}\_{dict}$ is equivalent to maximizing a lower bound of $I(\mathbf{h}, y)$, could a similar statement be derived for $I(\mathbf{h}, r)$ or $I(r, y)$?

Author response:
We firstly appreciate the Reviewer SF1W’s pointing out that the proof in Appendix B is contributing. As Reviewer SF1W’s feedback, similar statements for $I(\boldsymbol{h}, r)$ cannot be derived. Appendix B proves that minimizing $L_\mathrm{dict}$ increases the lower bound on $I(\boldsymbol{h},y)$. With the inequality used in [2],  $I(\boldsymbol{h}, r) <=  I(\boldsymbol{h}, \boldsymbol{x}) - I(\boldsymbol{h}, y)$,  $I(\boldsymbol{h}, r)$ will increase if $I(\boldsymbol{h},\boldsymbol{x})$ increase more than $I(\boldsymbol{h},y)$ increases. Likewise, $I(\boldsymbol{h}, r)$ will decrease if $I(\boldsymbol{h},y)$ increase more than $I(\boldsymbol{h},\boldsymbol{x})$ increases. Thus, to make a claim on $I(\boldsymbol{h},r)$, we should make a deductive claim on how $L_\mathrm{dict}$ affects $I(\boldsymbol{h},\boldsymbol{x})$.


Theoretically, making such a claim appears implausible, given that $L_{dict}$ pertains solely to the relationship between $\boldsymbol{t}_z$ and $\boldsymbol{h}$, and does not extend to the relationship between $\boldsymbol{h}$ and $\boldsymbol{x}$. Instead, Figure 4 empirically demonstrates a trend that $I(\boldsymbol{h}, \boldsymbol{x})$ progressively decreases as layers progress, resulting in the decline of $I(\boldsymbol{h},r)$.


In other words, we focused on the empirical aspect of $I(\boldsymbol{h}, r)$, demonstrating that our loss function is more effective in reducing $I(\boldsymbol{h}, r)$ compared to conventional contrastive loss in the absence of auxiliary networks $f_\phi$. Although the proof in Appendix B is vital to our paper, given the considerations mentioned earlier, please consider that we are unable to relocate it due to its considerable length.


However, we already know $I(r, y)=0$ by the definition of task irrelevant variable $r$: "any random variable that affects the observed data $x$, but is not informative
to the task we are trying to solve" such that $I(r, y) = 0$ [1,2]. Intuitively speaking, observing task irrelevant variable $r$ gives no insight about task-relevant variable $y$, and vice versa.

Reviewer comment (Experiments):
Do the learned label embeddings have any semantic meaning? The authors could verify this through dimensionality reduction on $\mathbf{t} \in \mathbf{D}^{Z}$, in the context of a dataset with a large number of classes (i.e., CIFAR-100).

Author response:
Many thanks for this interesting suggestion. The outcomes from your suggestion are indeed meaningful. As depicted in Figure 15, the t-SNE visualization of label embeddings post-training on CIFAR-100 (with 20 super-labels and 100 sub-labels) illustrates a noteworthy pattern. Labels within the same super-label group tend to cluster together. Furthermore, labels from different super-label groups are observed to be in close proximity if they share semantic similarities (e.g., the embedding for 'forest' is closer to that of 'trees' compared to other embeddings from the 'large outdoor scene' category.)”. We have incorporated a discussion of these findings into Section 5.3 of our paper.

Reviewer comment (Experiments):
One possible comment is that the architectures used are not exactly SOTA — it would be desirable to see this done with ResNets, ViT, RegNets, and so forth, as this could enhance the paper greatly.

Author response:
We thank Reviewer 3KwZ for the suggestion. Because we are more familiar with MLP-Mixer than RegNet, we test $L_\mathrm{dict}$ with ResNet-32, ViT, and MLP-Mixer in Appendix E. These architectures are composed of multiple modules, with each one acting as a functional unit. For example, in ViT, the multi-head attention (MHA) and MLP cooperate as a single unit (module); MHA is responsible for capturing attention, and the MLP is tasked with processing it. We maintain that these modules are integral to their respective architectures and should remain coupled, as decoupling them would disrupt their designed functionality. Consequently, in our experimental setup, we applied $L_{dict}$ at a module level, incorporating module-wise BP. However, we did not employ module-wise auxiliary networks used in LL. Moreover, we compare our results with $L_\mathrm{contrast}$ (as defined in Eq (1)), which does use such networks.


|         | ResNet-32        |         |             | ViT             |         |             | MLP-Mixer      |         |             |
|---------|------------------|---------|-------------|-----------------|---------|-------------|----------------|---------|-------------|
|         | Errs.            | Δθ      | Memory      | Errs.           | Δθ      | Memory      | Errs.          | Δθ      | Memory      |
| BP      | 6.701            | 0       | 3179 MiB    | 16.25           | 0       | 5300 MiB    | 17.23          | 0       | 6361 MiB    |
| $L_{contrast}$ | 32.71          | 73.6K   | 1617 MiB    | 32.95          | 394K   | 1354 MiB    | 31.67         | 394K   | 1468 MiB    |
| $L_{dict}$     | 25.19          | 5.12K   | 1617 MiB    | 32.63          | 5.12K  | 1348 MiB    | 23.51         | 5.12K  | 1445 MiB    |


The results, which are compiled in Table 8, indicate that our approach is not only more parameter and memory efficient but also yields superior results compared to $L_\mathrm{contrast}$ across all tested architectures.

Reviewer comment (Experiments):
It is interesting that the authors chose to have one embedding per class for all layers. Different layers can achieve different levels of clustering, and therefore both the embeddings and the loss weight on them can be different (see for example [2]). Another alternative would be to learn an MLP from the features to the embeddings rather than using average pooling. It is unclear whether the authors tried this direction.

Author response:
We express our sincere appreciation for Reviewer 3KwZ’s valuable suggestions. Following the advice, we have applied and evaluated the layer-wise dictionary $\boldsymbol{D}^Z_l$ across datasets. 
| Method     | MNIST | F-MNIST | CIFAR-10 | CIFAR-100 | SVHN  | STL-10 |
|------------|-------|---------|----------|-----------|-------|--------|
| DC-FL      | 0.33  | 5.52    | 8.64     | 31.75     | 2.19  | 22.87  |
| DC-FL-LD   | 0.34  | 5.50    | 8.45     | 31.64     | 2.19  | 22.59  |

The results indicate that the layer-wise approach outperforms a single dictionary overall. Moreover, Reviewer 3KwZ’s recommendation presents an effective method for parallel training contexts where each layer could be processed on a separate GPU.


We again convey our deep gratitude for  Reviewer 3KwZ’s insightful recommendations concerning alternatives to pooling. In an experiment based on this suggestion, we mapped $\boldsymbol{t}_z$ to $\boldsymbol{t}_z^l$ using a layer-wise fully connected layer $f_P$, which act as the auxiliary networks for $\boldsymbol{t}_z$. The results, as displayed in Table 9, were unexpected: $f_P$ led to a decline in performance for both $\boldsymbol{D}^Z$ and $\boldsymbol{D}^Z_l$. Thanks again to Reviewer 3KwZ’s suggestion. This experiment provides an ablation study on $pool_l$. 



|       | CIFAR-10     |           | CIFAR-100   |           |
|-------|--------------|-----------|-------------|-----------|
|       | Errs.        | Memory    | Errs.       | Memory    |
| $\boldsymbol{D}^Z_l$ | 11.70        | 770 MiB    | 33.15      | 785 MiB    |
| $\boldsymbol{D}^Z$   | 12.47        | 753 MiB    | 36.08      | 770 MiB    |
| $ pool_l $   | 8.64         | 747 MiB    | 31.75      | 751 MiB    |

Reviewer comment (Experiments):
It is unclear whether the authors considered having more than one embedding per class, given that there may be variability within each class. This could be treated as a hyperparameter: the number of embeddings per class.

Author response:
We express our deepest gratitude for  Reviewer 3KwZ’s valuable insights. Following  this suggestion, we have conducted an experiment to explore the concept of multiple embeddings per class, specifically in a scenario where we can more precisely discern variations within each class. Given that CIFAR-100 is organized into 20 super-classes, each comprising 5 sub-labels, we know exactly how each super-class label (super-label) varies within the 5 sub-labels, thereby obtaining 5 sub-labels per super-label. We tested models trained on CIFAR-100 to make inferences on 20 super-labels, using two methods, 1) assigning the super-label based on the closest proximity of the local feature $\boldsymbol{h}$ to any sub-label embeddings within the super-label (“Super”); 2) averaging the sub-label embeddings to create a unified super-label embedding, which is then used to predict the closest super-label during inference (“Mean”).
|       | FC    | Conv  | VGG   |
|-------|-------|-------|-------|
| E2E   | 56.65 | 40.72 | 23.42 |
| Naive | 65.85 | 49.78 | 31.75 |
| Super | 60.95 | 37.56 | 21.84 |
| Mean  | 60.98 | 48.61 | 32.26 |


The results, presented in Table 7 in the revised script, reveal that the "Super" method outperforms the others, while the "Mean" method does not improve upon the baseline approach (predicting sub-label's super-label as the prediction). We speculate that this is due to the presence of outlier sub-labels. For instance, as demonstrated in Figure 7 in the revised script, the "forest" sub-label is more closely aligned with the "trees" super-label rather than its own "large natural outdoor scene" group, skewing the average embedding for the super-label and causing a bias. Additionally, our results indicate that the model learns semantic relationships more effectively than dataset-defined taxonomies. For example, "chimpanzee" is categorized under
"large omnivores and herbivores" alongside "elephant, cattle, camel, and kangaroo," yet it more intuitively aligns with "people," as illustrated in Figure 7.

Reviewer comment (Novelty):
The paper relates closely to the literature on neural collapse in intermediate layers. It would be beneficial for the authors to mention these works and make the connection ([1-4, 6]).

Author response:
We extend our deepest thanks to Reviewer 3KwZ for insightful recommendations of significant literature. Our method aligns closely with the neural collapse phenomenon, particularly NCC or NC4 [1]. As [2,3,4] suggest, neural collapse also emerges in the intermediate layers. By minimizing $\mathbb{E}[L_{dict}(\boldsymbol{h},\boldsymbol{D}_Z)]$, our approach actively encourages the alignment between the average of local features with label $z$ and label embedding vectors $\boldsymbol{t}_z$, akin to the motivation of local losses used in [2,5].


However, our work is different from these studies in its exploration of locally decoupled contexts. Existing works [2,3,4,5] focus on local features within an end-to-end BP framework, whereas we investigate these features in scenarios where each layer or module functions independently. In this perspective, every local feature can be considered as the “final feature before the classifier”, which is where neural collapse emerges. In fact, as detailed in Appendix C, our methodology is capable of making layer-wise predictions. Figure 9 in the revised script demonstrates an increase in accuracy as we progress through the layers, resonating with observations made in [3].


This unique perspective allows us to uncover and understand the effects of neural collapse in a different and possibly more nuanced computational environment. We've drawn connections to neural collapse in our revised manuscript in Appendix F, even though it wasn't the central focus of our original paper, to elucidate the relevance of our findings within the broader landscape of neural network research.

Reviewer comment (Novelty):
There have been previous works that use intermediate layer losses to encourage class label clustering in the BP setting ([2, 5]).

Author response:
We extend our deepest thanks to Reviewer 3KwZ for insightful recommendations of significant literature. Our method aligns closely with the neural collapse phenomenon, particularly NCC or NC4 [1]. As [2,3,4] suggest, neural collapse also emerges in the intermediate layers. By minimizing $\mathbb{E}[L_{dict}(\boldsymbol{h},\boldsymbol{D}_Z)]$, our approach actively encourages the alignment between the average of local features with label $z$ and label embedding vectors $\boldsymbol{t}_z$, akin to the motivation of local losses used in [2,5].


However, our work is different from these studies in its exploration of locally decoupled contexts. Existing works [2,3,4,5] focus on local features within an end-to-end BP framework, whereas we investigate these features in scenarios where each layer or module functions independently. In this perspective, every local feature can be considered as the “final feature before the classifier”, which is where neural collapse emerges. In fact, as detailed in Appendix C, our methodology is capable of making layer-wise predictions. Figure 9 in the revised script demonstrates an increase in accuracy as we progress through the layers, resonating with observations made in [3].


This unique perspective allows us to uncover and understand the effects of neural collapse in a different and possibly more nuanced computational environment. We've drawn connections to neural collapse in our revised manuscript in Appendix F, even though it wasn't the central focus of our original paper, to elucidate the relevance of our findings within the broader landscape of neural network research.

Reviewer comment (Theory):
An important motivation of this work and LL/FL as a whole is the goal of removing backprop (BP), because BP is not biologically plausible and is not memory efficient. The abstract and intro of this submission make this point very clear. My question is: If you use a neural network $\phi$ in Eqn. (1), then how can you remove BP? Don't you use BP to train this $\phi$? It might be true that using a smaller network $\phi$ can save memory, but my point is BP is still there if you are using this $\phi$, unless you are updating this $\phi$ with local learning too (which I don't think is the case). Thus, I don't think the proposed method can be called forward learning (FL), which by definition should have no BP at all. And it is not very fair to compare the proposed method with FL methods.

Author response:
We are truly grateful to Reviewer UJZ4 for insights and understanding. While Reviewer UJZ4 has overall grasped our work well, we would like to clarify one particular aspect: we did not utilize auxiliary networks $f_\phi$ in our approach. As Reviewer UJZ4 pointed out, LL uses module-wise auxiliary networks and hence has module-level BP to update auxiliary networks. Yet, FL does not use auxiliary networks, being free from BP. Therefore, we can say that DC-FL is a FL framework. Our motivating experiments (in Section 3) starts from what happens if there is no auxiliary network (i.e., FL) in InfoPro [1], comparing $L_\mathrm{feat}$ (auxiliary network-free InfoPro) to $L_{\mathrm{contrast}}$ (InfoPro). This observation (Figure 1) motivated the need for a novel solution that is entirely independent of $f_\phi$ while still delivering competitive performance.

Reviewer comment (Theory):
In Eqn. (2), why there is an average over $K$? And why does there need to be $K$ feature maps for one class in the first place? Eqn. (2) can be simplified as $L_{dict} = -\frac{1}{N} \sum [ \text{log} \frac{\text{exp}( \langle h_n, t_+' \rangle )  }{ \sum \text{exp}( \langle h_n, t_i' \rangle ) } ]$, where $h_n$ is the average of $h_n^k$. So why not just use one $h_n$, but use K $h_n^k$ if they are equivalent?

Author response:
We agree your advice that $\frac{1}{K}\sum^K_{k=1}\langle\boldsymbol{h}_n^k, \boldsymbol{t} \rangle$ is mathematically equivalent to $\langle \boldsymbol{\bar{h}}_n, \boldsymbol{t}\rangle$, where $\boldsymbol{\bar{h}}_n=\frac{1}{K}\sum^K_{k=1}\boldsymbol{h}_n^k$. In fact, we used the simplified expression (what Reviewer UJZ4 suggested) for the mathematical proof in Appendix B. However, please consider that ‘averaging feature vectors first’ could degrade performance due to more severe loss in floating-point precision, which is typically more pronounced when handling smaller numbers. Averaging the dot products, which generally result in higher values, is less prone to precision loss. Conversely, averaging the feature vectors before the dot product can lead to greater precision loss, given that the values in feature vectors are usually smaller. Our intention was to accurately represent the implemented version of the loss. However, considering the simplicity of the expression Reviewer UJZ4 highlighted, we have revised the notation in Eqn. (2) and included a footnote to address the potential loss of floating-point precision.

Reviewer comment (Experiments):
My biggest concern with the experiments is that this work is built upon InfoPro [1], yet in the experiments it is not compared to InfoPro [1] at all. I feel that it is very likely that the method proposed in this work has a very similar performance to InfoPro.

Author response:
With the above-mentioned respective, though our proposed objective is quite similar to that of InfoPro in that both use contrastive loss, they are different from each other (LL vs. FL). Nonetheless, sincerely consenting with Reviewer UJZ4, we conducted a further comparison with InfoPro.
| Method     | MNIST | F-MNIST | CIFAR-10 | CIFAR-100 | SVHN  | STL-10 |
|------------|-------|---------|----------|-----------|-------|--------|
| LL-contrec | *0.65 | *5.71   | *9.02    | *31.35    | *2.34 | *29.74 |
| LL-cont    | *0.37 | *5.92   | *7.72    | *31.19    | *2.29 | *26.83 |
| DC-FL      | 0.33  | 5.52    | 8.64     | 31.75     | 2.19  | 22.87  |


| Method     | MNIST, F-MNIST |       | CIFAR-10, SVHN |       | CIFAR-100 |       | STL-10 |       |
|------------|----------------|-------|----------------|-------|-----------|-------|--------|-------|
|            | Δθ             | Memory Cost | Δθ             | Memory Cost | Δθ    | Memory Cost | Δθ    | Memory Cost |
| LL-contrec | 1.15M          | 811 MiB     | 2.07M          | 1049 MiB     | 2.07M | 1050 MiB     | 2.07M | 5954 MiB    |
| LL-cont    | 918K           | 695 MiB     | 1.84M          | 894 MiB      | 1.84M | 895 MiB      | 1.84M | 1846 MiB    |
| DC-FL      | 5.12K          | 580 MiB     | 5.12K          | 747 MiB      | 51.2K| 751 MiB      | 5.12K | 1589 MiB    |

Reviewer comment (Experiments):
The experiments compare the proposed method with FL methods, which as I said is not very fair, because I don't think the proposed method is FL since it still uses backprop when updating $\phi$. InfoPro [1] did not claim their method to be FL. In fact, their paper did not mention FL at all.

Author response:
We are truly grateful to Reviewer UJZ4 for insights and understanding. While Reviewer UJZ4 has overall grasped our work well, we would like to clarify one particular aspect: we did not utilize auxiliary networks $f_\phi$ in our approach. As Reviewer UJZ4 pointed out, LL uses module-wise auxiliary networks and hence has module-level BP to update auxiliary networks. Yet, FL does not use auxiliary networks, being free from BP. Therefore, we can say that DC-FL is a FL framework. Our motivating experiments (in Section 3) starts from what happens if there is no auxiliary network (i.e., FL) in InfoPro [1], comparing $L_\mathrm{feat}$ (auxiliary network-free InfoPro) to $L_{\mathrm{contrast}}$ (InfoPro). This observation (Figure 1) motivated the need for a novel solution that is entirely independent of $f_\phi$ while still delivering competitive performance.