Paper: FeatUp: A Model-Agnostic Framework for Features at Any Resolution (ICLR-2024)

Reviewer comment (Experiments):
The focus of the paper seems unclear. Being a feature upsampling framework, the framework should be agnostic to the specific downsampling or upsampling operator chosen. Yet, the current approach leaves me an impression that the framework only works on the proposed learned downsamplers and upsamplers. If the framework matters, some additional existing downsampler-upsampler pairs should be demonstrated to demonstrate its generality; If the proposed operators matter, the contributions and the title may need to be revised.

Author response:
**Justifying the focus of the paper**

We present FeatUp as a framework that generalizes across backbones and upsamplers. The JBU and Implicit variants are our novel upsampler contributions that outperform existing upsamplers, but we do evaluate others. In particular:
- An important aspect of generality is the fact that FeatUp can work with any backbone network (see Fig. 6), making it applicable to many models in the community.
- Though we present our novel JBU upsampler that improves performance, Table 1 shows 4 other upsamplers can be trained with FeatUp including implicit networks, resize-convs, CARAFE [1], and SAPA [2]. These upsamplers are all trained with our proposed scheme (besides bilinear, strided, and large-image methods, which aren't trained). We note that our JBU variant, however, provides improvements over all other upsamplers. 
- Though we provide 2 example downsamplers, we stress that this component is primarily dictated by “nature”. To draw a direct analogy with NeRF [3], the downsampler is analogous to ray-marching, which approximates the physics of light. In NeRF, this component can be modified slightly, (lens distortion, surface properties, light fields, etc) but needs to approximate physics. For FeatUp, Figure 6 shows that modern networks pool/blur information which leads us towards pooling/blurring operators for the downsampler. We also elaborate on this in the global response section “Clarifying the role of the saliency maps”. We have justified these downsamplers with extensive experiments, visualizations, and ablations.

Reviewer comment (Novelty):
Following my previous comment, I think the problem is that the authors attempt to combine many seemingly related techniques together. For instance, the JBU and its fast CUDA implementation look standalone to me. It is unclear to me what the role of the JBU is in the framework, particularly when it is claimed as a contribution. Please explain why JBU is essential to the FeatUp framework. In addition, applying JBU to feature upsampling is not new. "Fast End to End Trainable Guided Filter" CVPR18 and "Superpixel Convolutional Networks using Bilateral Inceptions" ECCV16 are two example works, which is also missing in the related work.

Author response:
**Role of JBU**

Our paper focuses on making a system that can efficiently upsample features and retain their original semantics. We find that existing upsampling layers do not properly utilize guidance information as well as the JBU does. Our analysis shows our JBU operator beats all existing upsamplers for creating upsampled features, which is why we investigated it. We hope you can see this as an additional novelty of our work. 

**Explaining our relationship to “Fast End to End Trainable Guided Filter”**

We thank the reviewer for bringing this work to our attention and have added it to our related work section. We note that “Fast End to End Trainable Guided Filter” [12] does not perform the full JBU computation but instead approximates it with learned pointwise convolutions and linear maps because the local query/model was computationally intractable. This is precisely the problem we solve exactly with our new efficient CUDA kernel.

Reviewer comment (Experiments):
The ablation study of the paper is poor. It is unclear to me what exactly leads to the good feature quality. Aside from the loss, there are a number of tricky designs in the downsampler and upsampler used. For example, why a blur kernel must be used in downsampling? And why introducing the saliency map and how it works? Why a simple 1x1 convolution is sufficient to present the saliency map? Would it be precise enough? Also, the implicit FeatUp produces significantly better visualizations according to Fig. 5. Is it because of the per-image optimization. A thorough ablation study is required.

Author response:
**Quantitative Ablations**

We added ablations of every major architectural decision of FeatUp in the global response. Each ablation degrades performance across the metrics considered.

Reviewer comment (Experiments):
The method may be justified in an inappropriate setting. According to Table 1, FeatUp (Implicit) has reported significant performance improvements on the transfer learning setting. Yet, this setting is rarely used in feature upsampling literature. Why choosing this setting? I wonder how the parameters of comparing upsamplers are updated in this setting. In addition, the comparison between the implicit variant and other upsamplers seems unfair, because other upsamplers do not optimize the per-image upsampling quality. This also leads to the next issue.

Author response:
**Justifying the experimental settings**
- We use existing experimental settings from the literature (FADE [4] + SAPA) for Table 2 
- We compare against other per-image optimization methods including Deep Image Priors (DIP) [5] in Table 2 and Figure 8 and ZSSR [6] in Figure 8.
- Transfer learning is an established setting to measure feature quality. It is used throughout the unsupervised representation learning literature including in methods like STEGO [7], PiCIE [8], MOCO [9], SimCLR [10], DINO [11] and others. This analysis also shows we improve performance on downstream tasks *without retraining* downstream components, which is important for practitioners who want to upgrade or better understand existing methods. 
- The parameters in the upsampler comparison of Table 1 are updated using the FeatUp training losses. We use these upsampling operators instead of JBU demonstrating that our framework works with a variety of upsamplers.

Reviewer comment (Experiments):
The practicality of the implicit FeatUp seems poor. In reality, the per-image optimization of upsampling is unrealistic, it is not likely that the implicit variant is used as a plug-in in existing networks (the experimental setting in Table 2 also confirms my opinion). But I think this approach can somehow indicate an upper bound of upsampling quality.

Author response:
**Justification of FeatUp (Implicit) method**

Please refer “Practicality of FeatUp (Implicit)” in the global response for advantages of any-resolution features. FeatUp (Implicit) is not the only per-image-optimized method as we compare to the widely-used approaches DIP and ZSSR that optimize results per image.

Reviewer comment (Experiments):
The performance of baseline approaches does not align with the number reported in the corresponding papers and is significantly lower. For example, SAPA reports 44.4 mIoU but the table in Table 2 only reports 41.6. Please check.

Author response:
**Performance discrepancy of SAPA**

We appreciate you pointing this out. We have used SAPA’s official code, and use the encoder-decoder architecture and training objective detailed in SegFormer [13] and followed by FADE evaluations. All experiments are performed using the same controlled settings and environment to ensure a fair comparison between methods. We note this discrepancy the paper for full transparency.

Reviewer comment (Presentation):
The organization of related work does not make sense to me. First, the paper does not conduct experiments on image super-resolution. This sub-section seems redundant.

Author response:
**Organization of Related Work**

We have incorporated all suggested works into the related work. We evaluate some image super-resolution methods (DIP, ZSSR, and LIIF) in Section 6.3 which is why we included them in our related work. We shortened this section and directed readers to the supplement to provide more context. Regarding Amir et al. (2021) and Tumanyan et al. (2022), both works describe a modified patch embedding stride which is our “strided” baseline.

Reviewer comment (Experiments):
A main goal of upsampling the feature map is to use it for tasks that requires semantic understanding and dense prediction, e.g. semantic segmentation. How does the proposed method work for more standard semantic segmentation benchmarks (for both mIoU and computation cost) and what are the main advantages of the proposed method? For example, in the context of any-resolution upsampling of feature map, Learning implicit feature alignment function for semantic segmentation (ECCV 2022) seems to be related, does the proposed method show advantages over the prior work?

Author response:
**Segmentation Benchmarks**

We evaluate commonly-used upsamplers on the SegFormer [1] architecture and the ADE20k [2] dataset, which is a standard semantic segmentation benchmark. As seen in Table 2, FeatUp outperforms the other upsamplers in mIoU, mAcc, and aAcc while remaining competitive in parameter count (it uses fewer parameters than every upsampler meant for features, with only A2U being lighter - however, A2U was designed for image matting and does not perform well on this task).

**Comparison to additional references**

We thank the reviewer for pointing out related prior work on feature upsampling, and have added them in our related work section.

We have also implemented “Learning Implicit Feature Alignment Function for Semantic Segmentation” [3], and added a visualization to Figure 8. While IFA performs well on specific semantic segmentation benchmarks, it does not take advantage of image guidance and fails to learn high quality representations outside of the encode-decoder framework.

Reviewer comment (Evaluation):
How does the computation cost (in FLOPS) compare to prior works in main tables? Since the proposed method is a new upsampler, it would better demonstrate the advantages of the proposed method by comparing the computation cost.

Author response:
**Computation Cost Analysis**

We analyze how GFLOPs (= FLOPs * 1e-9) scale for each upsampling operation in a new figure (see Figure 16 in the paper) and we add an analysis of FLOPS to Table 2. In general we find that FeatUp’s JBU operator is lightweight compared to other upsamplers and is small compared to the cost of evaluating the backbone network.

In particular, we study the original CARAFE [4], a “guided” CARAFE (the original operation with the guidance concatenated with the source to mirror the other guided operations in this study), SAPA [5], FADE [6], and FeatUp (JBU). Besides the attribute being varied (upsampling factor, feature dimension, or target spatial dimension), all other experimental settings are kept constant and exemplify a realistic feature-upsampling scenario (detailed below).

- First, we vary the upsampling factor. In each experiment, we begin with a $(1, 256, 8, 8)$ tensor, guide it with a $(1, 256, 8n, 8n)$ tensor when guidance is appropriate, and produce a $(1, 256, 8n, 8n)$ tensor with $n \in \{ 2, 4, 8, 16, 32 \}$. 
- In the second plot, we vary the channel dimension. In each setup, we begin with a $(1, c, 8, 8)$ tensor, guide it with a $(1, c, 16, 16)$ tensor when guidance is appropriate, and produce a $(1, c, 16, 16)$ tensor with $c \in \{ 32, 64, 128, 256, 512 \}$.
- Finally, we vary the spatial dimension of the output. In each setup, we begin with a $(1, 256, s//2, s//2)$ tensor, guide it with a $(1, 256, s, s)$ tensor when guidance is appropriate, and produce a $(1, 256, s, s)$ tensor with $c \in \{ 16, 32, 64, 128, 256 \}$.

Note that the resize-conv operation’s GFLOPs can also be calculated in the same way, but when visualized on these plots, its computational load grows so quickly that the upsamplers depicted appear to have constant performance. Thus, we show the upsamplers that are most competitive in performance.

We find that FeatUp is competitive in GFLOP count to the current leading upsamplers and is lighter weight than FADE. It is comparable to CARAFE’s computational cost. These GFLOP analyses show that FeatUp (JBU) is not only architecturally and qualitatively a reasonable substitute for existing upsamplers, it is competitive in # FLOPs across a wide array of scenarios.