Paper: Sub-token ViT Embedding via Stochastic Resonance Transformers (ICLR-2024)

Reviewer comment (Experiments):
Can SRT be applied to models beyond Transformers? It would be of interest to see empirical results exploring its applicability to other models such as ResNets or MLPs.

Author response:
**Q1:** *Can SRT be applied to models beyond Transformers? It would be of interest to see empirical results exploring its applicability to other models such as ResNets or MLPs.*

**R1:** As stated in the paper, 'It is worth noting that Stochastic Resonance is not limited to ViT architectures nor to spatial quantization. It can be applied to architectures like CNN as well as to other forms of quantization, such as sale or domain size. However, our emphasis in this paper is on ViTs that mostly use non-overlapping tokens, making them particularly suited to our approach.' (Section 4.2)

As suggested by the reviewer, we applied SRT to ResNet on classification problem on Cifar, the results are in the Appendix A, where we improve classification accuracy by an average of 5.87%.

**Supervised classification on Cifar-10**
Architecture | ResNet20 | ResNet32 | ResNet56
| ----------- | ----------- | ----------- | ----------- |
| Accuracy | 91.95 | 92.68 | 93.50 |
| Accuracy w/ SRT | **92.41** | **93.14** | **93.87** |
| Relative Error Reduced | 5.6% | 6.3% | 5.7% |

Reviewer comment (Experiments):
It is important to understand the hyper-parameters involved and perform in-depth analysis when noise level can be effective or harmful. Since stochastic resonance may not be suitable for all cases, how to determine the optimal noise levels in practical applications?

Author response:
**Q2:** *It is important to understand the hyper-parameters involved and perform in-depth analysis when noise level can be effective or harmful. Since stochastic resonance may not be suitable for all cases, how to determine the optimal noise levels in practical applications?*

**R2:** That's a great question, and admittedly an ongoing effort to study how to best determine them outside of empirical trials. In our experiments, we noticed that using an augmentation level close to one-fourth of the patch size works effectively for various tasks. We look forward to future research shedding light on this question.

Reviewer comment (Experiments):
It would be helpful to analyze the noise distribution and perform experiments using other augmentation strategies.

Author response:
**Q3:** *It would be helpful to analyze the noise distribution and perform experiments using other augmentation strategies.*

**R3:** We appreciate the reviewer's input but may need further clarification on the term 'noise distribution' that the reviewer is referring to. If the reference is to artificial noise added as augmentation for stochastic resonance, then indeed, various options are available, as long as they are group transformations which guarantees an inverse transformation. However, we specifically chose translation, as it avoids introducing interpolation artifacts, unlike rotation, and provides finer control over the ensemble (pixel displacement in rotation depends on its distance from the center). Importantly, for transformers with positional encodings, flipping is not a viable option since the embeddings are not designed to be invariant.

If the term 'noise' pertains to stochastic noise in the features, assuming the ensembled features represent a "denoised" signal, we measured it by the Euclidean distance between the ensembled feature and the feature from a single forward pass, aggregated as a histogram. We show this noise distribution in Fig. 4 in the revised manuscript's Appendix.

Reviewer comment (Experiments):
Why not conduct experiments on the main task in computer vision such as image classification, semantic segmentation, and object detection?

Author response:
**Q4:** *Why not conduct experiments on the main task in computer vision such as image classification, semantic segmentation, and object detection?*

**R4:** We focus on zero-shot tasks to highlight the capabilities of SRT without any fine-tuning. However, as explained in the paper, SRT can be used for any layer in any Vision Transformer (VIT). In response to the suggestion, in addition to the classification results presented in Q1, we present results for semantic segmentation in the Appendix B. SRT consistently improves mIoU on all three pre-trained ViTs, by as much as 1.7% in relative improvement. A discussion is provided in the Appendix:

**Semantic segmentation by DinoV2 with linear head on ADE20K**
Architecture | baseline | d=1 | d=2 | d=3
| ----------- | ----------- | ----------- | ----------- | ----------- |
| ViT-S/14 | 44.24 | 44.44 | 44.57 | **44.64** |
| ViT-B/14 | 47.28 | 47.63 | 47.85 | **47.98** |
| ViT-L/14 | 47.49 | 48/18 | 48.44 | **48.62** |

Reviewer comment (Experiments):
Perturbations are translation only. Translations are one of the perturbations, and can be replaced by a simple convolution kernel. If translation based perturbation works, then it's possible that other complex perturbations like rotation should also work. Also, since the translation works, it seems it can be replaced by an convolution network, followed by some deconvolution kernels in the upsampling stage, with some conv layers to do the aggregation. Then it will become a learnable model instead of translations only. In this way the model can not only simulate the translation, but also other non-linear perturbations.

Author response:
The reviewer is correct that we could replace the translated pillbox kernel we use (although we do not explicitly mention) with any other kernel undergoing any other group (invertible) transformation. Those would give rise to generalized forms of convolution such as Fourier-Mellin etc., which indeed could further improve our method if the sampling of the group is adapted to the signal. We simply use the most obvious choice (pillbox = constant) of kernel and of group (translation) for simplicity. 

Moreover, from an engineering perspective, employing translation as a group transformation offers several advantages:
Assuming pixel displacements in the translation matrix, one can map every feature to discrete pixel locations, which avoids interpolation artifacts commonly associated with other transformations, i.e., rotation, scaling.
Translation does not change the size for objects in the image, resulting in more stable features than scaling.
Unlike rotation, where pixel displacement varies based on distance from the center, pixel displacement of each patch can be directly controlled by specifying translation components in the matrix. 
We appreciate the reviewer's valuable suggestions for future work. However, it's essential to note that in this study, translation performs well and is validated by improvement in five different tasks.

Reviewer comment (Experiments):
Did not compare with image super-resolution method. Image super resolution is an well-studied area, and it's an natural thought to try some of them and see if the generated super-resolution features can help to improve the down-stream tasks.

Author response:
(Please also see the "general message".) We wish to emphasize that ours is an ensembling method, not a super-resolution method in the traditional sense: We capture multiple samples (feature maps) from the original signal (image), which span a sigma-algebra strictly containing that of a single sample, from which one cannot retrieve the information lost simply by post-processing. We could use an analogy with acoustic processing where super-resolution would be akin to noise reduction, whereas stochastic resonance is threshold reduction. There may be a misunderstanding in the term “super-resolution”, in the sense that we do not increase the size, but enhance the detail. We will revise the text to make this clear.

Reviewer comment (Experiments):
Is it possible to try other image super resolution models and see how the performance looks like? Theoretically, if the simple resizing works, then other super resolution should work better.

Author response:
Regarding super-resolution, please see the [general message]. 

In practice, the use of image super-resolution on the features is not feasible because, unlike the standard 3 channels (RGB), low-resolution features typically have many more channels (e.g., 256 for ViT/S or 768 for ViT/B). Existing super-resolution methods are not designed to handle inputs with such a high number of channels. We acknowledge the reviewer's concern and conducted experiments by resizing, some results are highlighted below (for a full table, see Table 7 in the revised paper). Overall simple resizing is detrimental to the task. This may be because there is no spatial smoothness constraint that is imposed on ViT architecture or training process, so a simple resizing operation introduces out-of-distribution features to the network.

**Feature interpolation on monocular depth estimation with NYU-V2 and DinoV2**

Architecture | Method | RMSE | RMSE_log | AbsRel | SqRel
 ----------- | ----------- | ----------- | ----------- | ----------- | ----------- 
| ViT-S/14 | baseline | **0.336** | **0.114** | **0.080** | **0.048** |
| | interpolation | 0.573 | 0.178 | 0.146 | 0.125 |

Reviewer comment (Experiments):
Is it possible to use some convolution kernels to replace the predefined perturbation type?

Author response:
In theory, perturbations are not restricted, as long as they adhere to group transformations, and translation can indeed be defined using convolution. However, in practice, finding more effective kernels remains an open problem, requiring further research. This paper concentrates on zero-shot tasks, where no additional training is performed. Nevertheless, we acknowledge that learned augmentations (so long as they are invertible) could be an interesting idea for certain specific tasks.

Reviewer comment (Writing):
The problem the paper seeks to tackle, while somewhat intuitively reasoned, seems insufficiently motivated. The quantisation artefacts are presented as a result of discrete partitioning into uniform square patches, but the exact nature of these artefacts as well as their effect on predictions are hardly discussed or touched upon. This makes the problem statement more ambiguous than necessary. Edit: The last revision addresses this with a formalisation of the context of SRT.

Author response:
We are not sure about the meaning of the term “exact nature”, but for imaging data the two most salient phenomena are occlusions and scale. Neither are respected by spatial quantization: occlusions are blended in samples that straddle the boundary, depriving the representation of precious information about the topology of the scene. Scale is also critically lost since the same spatial quantization of the image may correspond to any portion of an object, depending on its distance. This makes it difficult for the learned representation since it is forced to learn that the same object can span anywhere from a single sample to the entire image. 

In practice, the shortcomings of quantization schema are evident in DINO-based methods for segmentation, e.g. Tokencut, which only creates low-resolution segmentation due to quantization, and requires post-processing and refinement of pixel-level segmentation maps. 

This seems to be sufficient motivation to us, but if there are other aspects of spatial quantization that we missed we are certainly open to elaborate. We are more than happy to revise the paper to make the motivation clear to the readers.

Reviewer comment (Evaluation):
While the upscaled feature visualisations show improvement on the low resolution visualisations of the base model, the main concern on how useful the visualisations are as attributions for model predictions is not clear, despite the authors expanding their experiments with alternative interpolation methods. The authors revise the manuscript and no longer claim this as one of the main contributions of the work.

Author response:
We display the feature map only as a way of visualizing the outcome of SRT, which is a feature map that is strictly more informative than the original quantized map. It is difficult to determine whether feature maps are  “faithful” (and to what degree of “faithfulness”), so we measure it by the improvement on downstream tasks relative to the baseline. This is particularly the reason for conducting our experiments under zero-shot settings: Given that the encoding and decoding layers are frozen, features that are not faithful to the original would disrupt the outcome. On the contrary, our experiments show that the ensembled features yield consistent improvement for a number of downstream tasks including video object segmentation, depth prediction, unsupervised saliency segmentation, image retrieval and unsupervised object discovery; while it is difficult to define “faithfulness”, experimental results imply that the ensembled features are not “unfaithful”. 

Moreover, please refer to the Appendix of the revised manuscript for a comparison with (bilinear) interpolation. We compare SRT and simple interpolation to features from single forward passes. It is evident that SRT makes adjustments to the feature map at semantic boundaries, whereas simple interpolation displays a less meaningful grid pattern. We hope that this provides further support for the 'faithfulness' concern raised by the reviewer.

Reviewer comment (Novelty):
While the authors mention the conceptual overlap with Amir et al. (2021), this connection is not touched on to any further extent in the paper, except for a discussion on computational complexity. It seems natural to contrast against the results in this work since, by the authors' own admission, there is an inherent similarity between approaches. The authors expand the link in their revised work with additional results.

Author response:
Indeed, Amir et al. is only conceptually related and direct comparisons cannot be made: We could compare Amir et al. to our result prior to aggregation, but that would defeat the purpose because our goal is to retain the same dimension of the original representation. Amir et al. requires making modifications to the forward pass by using a stride smaller than the quantization domain to produce overlapping tokens that ultimately yields 2x increase in spatial resolution (but with 4x more tokens); whereas, our method does not require any changes to the forward pass and produces 16x increase in signal resolution. To match the same granularity, Amir et al. would need to produce 256x more tokens (65536x more compute), which makes it computationally infeasible. Nevertheless, we reproduced the results of Amir et al. and tested their 2x super-resolved ViT-S/16 feature maps on video object segmentation (Table 1, main paper) following protocol specified in DINO. Their features resulted in worse performance than the baseline pretrained model, whereas with just 1 pixel of displacement, SRT improves over the baseline (Fig. 3, main paper).

Reviewer comment (Reproducibility):
Methodology for non-dense downstream tasks (particularly image retrieval) seems unclear, and hardly reproducible. Several questions remain unanswered for practitioners wanting to reproduce the results in the paper. Note on resolution was addressed. While the authors seek to address gaps in methodology by releasing their code, more exposition in the manuscript would be ideal, but is further addressed in the last revision.

Author response:
Reproducibility will be facilitated by open-sourcing our code which we will do upon completion of the review process. In the meantime, We have added details about input resolutions as requested by the reviewer in our revised manuscript. For evaluation on retrieval, we follow protocols used in DINO and use the original resolution from the datasets.

Reviewer comment (Writing):
The link to stochastic resonance, while somewhat clear to this reviewer, is a little opaque. While this analogy could provide an intuitive understanding of the method, it is imperative to clarify the link and show the extent to which quantisation artefacts inhibit ViTs. Stochastic resonance typically involves the enhancement of weak signals in the presence of noise. In this context, techniques are applied for extracting super-resolved features by treating the discrete boundaries imposed by the partitioning in the tokeniser as the thresholds. In this reviewer's opinion, the link should be emphasised and clarified to improve the overall contribution of the paper. The authors agree that this is an issue, and have substantially revised their manuscript.

Author response:
This is a great suggestion. We will emphasize as suggested

Reviewer comment (Novelty):
While the method is novel, the current applications seem moderately niched.

Author response:
Respectfully, we disagree: In the main paper, we demonstrated five diverse applications of SRT: video object segmentation, depth prediction, unsupervised saliency segmentation, image retrieval, and unsupervised object discovery. We have additionally added classification and semantic segmentation in Table 5 and 6 of the Appendix, respectively, for a total of seven tasks covering a broad range of computer vision applications. SRT shows promise for all of them. 
 
We acknowledge the reviewer's perspective, as our emphasis on SRT's ability to achieve fine-grained feature maps may have been more pronounced. In response to this feedback, we plan to revise the paper to give greater emphasis to the practical applications of SRT.

Reviewer comment (Writing):
What is the precise nature of the artefacts the authors wish to remedy? Are the authors arguing that the low resolution PCA / attention maps exhibit these artefacts by virtue of having low resolution? This was addressed in the last revision.

Author response:
Spatial quantization causes information loss about key phenomenologies such as occlusion and scale, which we expect to impact performance in geometric and semantic inference tasks. Therefore, we expect that the artifacts of quantization will be manifest in all downstream visual tasks, and we test this assumption on different vision tasks.

Reviewer comment (Evaluation):
Have the authors considered adding metrics for evaluating the faithfulness of the attributions in the visualisations of the features? The authors downplay the contribution of the visualisations in their contributions, addressing this concern.

Author response:
Visualization is only for illustrative purposes, since the underlying representation is high dimensional. We use downstream metrics to evaluate the faithfulness of the representation to the content of the scene, despite quantization of the image. If there are other metrics thats evaluate the faithfulness we are also happy to include in the revised paper.

Reviewer comment (Reproducibility):
Exactly which embeddings were used for the KNN in the image retrieval task? The ensemble of class tokens? Pooled features? This is very unclear. The authors address this in their revision, however the methodology for non-dense prediction tasks is still unclear.

Author response:
We follow the protocol of the original DINO paper, which applies retrieval on the class token. Note that in order to obtain ensemble class token with SRT, we do not naively average the class token of augmented image, but ensemble PRIOR to the attention mechanism in the last layer. In this way the final class token is computed from the ensemble SRT feature. We have revised the paper to add the details as requested by the reviewer.

Reviewer comment (Evaluation):
The metrics for results in table 4, are they mAP? What does the columns 1-6 refer to? The authors address this in the revision.

Author response:
Yes mAP. 1-6 refer to noise level in SRT. We have revised the paper to make it clear to the readers.

Reviewer comment (Experiments):
The scores with salient segmentation using TokenCut omit comparisons with post processing using the bilateral solver proposed in the original paper. The reasoning behind this choice seems unclear. The authors addressed this by expanding their results.

Author response:
Our response regarding postprocessing: It's important to emphasize that our goal is to utilize TokenCut as a metric for comparing the original ViT feature with the ensemble SRT feature, rather than enhancing TokenCut itself. While post-processing techniques like Bilateral Solver or CRF refine segmentation boundaries, they don't directly address the comparison. As stated in the paper 'We execute TokenCut without any post-processing, such as Conditional Random Fields (CRF), to assess the raw quality of ViT embeddings.' (Sect. 3.4) Furthermore, our experiments are conducted fairly, using pooled features that share the same resolution as the original features employed by TokenCut.
Nevertheless, as requested by the reviewer, below we provide comparison of TokenCut with Bilateral Solver and ViT-S/16:

Datasets | Method | maxF | IoU | Accuracy |
----------- | ----------- | ----------- | ----------- | ----------- | ----------- |
ECSSD | Baseline | 87.4 | **77.2** | 93.4 |
ECSSD  | SRT | **88.4** | 77.0 | **93.6** |
DUTS | Baseline | 75.5 | **62.4** | 91.4 |
DUTS  | SRT | **76.5** | **62.4** | **91.7** |
DUTS-OMRON | Baseline | 69.7 | 61.8 | 89.7 |
DUTS-OMRON  | SRT | **70.6** | **62.4** | **89.9** |

We hope this answers the reviewer's question. If there are further concerns, we are more than willing to provide additional explanations.