Paper: Towards Faster and Stronger Deep Earth Mover's Distance for Few-Shot Learning (ICLR-2024)

Reviewer comment (Evaluation):
Limited discussion on the limitations of GEMD: The paper does not provide a comprehensive discussion on the limitations of the proposed GEMD method. It would be beneficial to include a section discussing the potential drawbacks or scenarios where GEMD may not perform as well, providing insights for future research and potential improvements.

Author response:
Thanks for the comment. In the last paragraph of Section 6 titled "Conclusion and Limitations" of the original manuscript, we briefly introduce drawbacks of GEMD. **Following your suggestion, we add separately a new section titled "Limitations and Future Research", discussing more comprehensively the limitations of GEMD. Kindly refer to Section A.7 of the modified paper.** The discussion is also attached below for ease of reference.

In this paper, we assume deep features follow Gaussian distributions. Despite the facts that Gaussian is of maximal entropy for given mean and covariance and has closed form distance, this assumption may not be optimal or even do hold. Depending on various architectures of deep neural networks (DNNs), feature distributions may be multimodal (multiple peaks) and, if single-peaked, may be different functions other than Gaussian. As far as we know, distribution of deep features is inadequately studied and future research on this topic can potentially further benefit representation learning in the few-shot regime. 

Our GEMD is not applicable to scenarios where distributions are  discrete, e.g., categorical distribution or  histogram; in these cases,  DeepEMD is a natural option. In addition, GEMD only focuses on classifier attached at the end of DNNs without touching on middle layers; in contrast, recent state-of-the-art methods design task-/domain-specific adapters, i.e., auxiliary lightweight sub-networks, for tweaking the middle layers of the backbone. GEMD is parallel to such methods and it is promising to combine them for further improvement.

Reviewer comment (Evaluation):
Lack of visualization and analysis: The paper does not provide any visualizations or in-depth analysis of the results. Including visualizations and further analysis of the learned features or matching flows would provide a deeper understanding of the proposed GEMD method and its effectiveness.

Author response:
Thank you for the concern. In Section A.3 titled "Visualization of GEMD" of the original paper,  we visualize the effect of patch matching of GEMD-M and compare with DeepEMD in 1-shot setting. Specifically, we provide visualization of matching of typical matches for four categories, i.e., mushroom, arctic fox, school bus and beer bottle. In most cases, GEMD shows better feature matchings than DeepEMD in few-shot learning. The visualization results show that the matching flows determined by GEMD have better capability to discriminate between the query and support images. Kindly see Section A.3 for details. 

**In terms of your suggestion, in Section A.3 of the modified paper, we add additional analysis for visualizing the transfer learning based GEMD-T through commonly used heatmaps.** 

For ease of reference, we also give brief discussion on visualization by heatmaps in the following; kindly refer to Section A.3 for complete visualization results and discussion.

We visualize the heapmaps of images by using CAM (Zhou et al. ,2016). We select four categories, i.e., Tench, Daisy, Rosehip and Meekat, for visualization with models learned by GEMD-T and DeepEMD.  On the whole, our GEMD can accurately focus on the regions of the objects while neglecting the backgrounds; in contrast, DeepEMD has larger tendency to be distracted by backgrounds. The comparison suggests that GEMD can learn more discriminative features than DeepEMD, demonstrating better capability of representation learning.

Reviewer comment (Novelty):
The conclusions in the paper are directly borrowed from the conclusions of previous articles, especially GEMD-T, where the derivation of some essential steps is missing, leading to a lack of coherence.

Author response:
Thanks for the concern. **In terms of your suggestion, for GEMD-T, we add separately Section A.6, providing essential steps for the proof. Kindly see the modified manuscript for details.**

Reviewer comment (Experiments):
Why wasn't the matching cost 1-cos(x, y) from DeepEMD used, and L2 distance was chosen instead? Does using 1-cos(x, y) still yield a more concise expression for EMD?

Author response:
As far as we know, except L2 distance, for other distances such as L1, Lp (p>2) and 1-cos(x,y), Gaussian EMD has no closed form expression. This is the reason why we choose L2 distance. EMD between Gaussians with 1-cos(x,y) as the cost may improve L2 distance, whose solution, however, is an open problem that requires theoretical investigation.

Reviewer comment (Experiments):
Can EMD be employed for the distillation learning of teacher-student models? If so, should using EMD for result distillation be considered in URL?

Author response:
Thanks for the questions. EMD in discrete form can handle categorical distribution, so can potentially be used in prediction (logits) based distillation learning, instead of the classical KLD. Gaussian EMD (GEMD) may be used in intermediate layers for knowledge transfer from features. In the framework of URL where a universal model is distilled from separately trained multiple models, EMD and GEMD can potentially replace KLD and CKA for transferring multi-domain knowledge, respectively.

Reviewer comment (Theory):
As explained in Sec.4.2, the method averages statistics from different samples to estimate the prototype. I wonder the difference between such implementation and treating all local descriptors from the same class as random samples from one Gaussian distribution and then estimating the statistics.

Author response:
Thanks. We implement the strategy you suggested to compute the prototypical Gaussian, called Strategy B for convenience. We compare it with our original method (Strategy A) in SDL setting on Meta-Dataset. Strategy B underperforms Strategy A for In-Domain (In-D) tasks while outperforming for Out-of-Domain tasks (Out-D). Overall, they perform comparatively. 

|Strategy | Method |In-D | Out-D | All | 
|:--------:|:--------:|:---------:|:---------:|:----------:|
|A|  Average samples  |  73.2    | 70.0    | 72.0    |
|B| All local descriptors  |  72.1    | 70.9    | 71.6    |

Reviewer comment (Theory):
The authors mention in the paper that KL divergence is not suitable because it is asymmetric. What about Jensen-Shannon divergence?

Author response:
Note that Jensen-Shannon divergence (JSD) between two Gaussians has no closed from expression (Nielsen, 2019). Hence, we compare with geometric Jessen-Shannon divergence (G-JSD) and dual G-JSD recently proposed in Nielsen (2019) that can be computed in closed form. G-JSD and dual G-JSD achieve comparable accuracies, both perform on par with ADM. Compared to our GEMD, they significantly underperform by large margins. 

|Method|Dis-similarity|In-D|Out-D|All| 
|:-|:-|:-:|:-:|:-:|
| GEMD-M | EMD | 62.0 | 72.8 | 72.0 |
| GEMD-T | EMD | 64.2 | 76.5 | 75.6 |
| ADM | KLD | 59.2 | 69.8 | 69.0 | 
| Ours | G-JSD | 60.4 | 69.5 | 68.8 |
| Ours | Dual G-JSD | 60.0 | 70.0 | 69.2 |

**Details on theory, implementation and discussion of (dual) G-JSD are given in the modified paper (cf. Section A.5.2).**

Reviewer comment (Experiments):
As for the efficiency comparison in Tab.3e, I think it would be better to compare the proposed method with not only EMD, but also other methods using pretrain and meta-train pipeline.

Author response:
Following your suggestion, we compare with state-of-the-art TSA, together with a commonly used baseline of ProtoNet. Compared to TSA, our GEMD-T is 2 times faster and meanwhile performing better.  ProtoNet consists of  pre-training plus meta-training along with meta-test where no training is required. Hence, ProtoNet runs the fastest, but with significantly inferior performance.  

| Method | Acc (%) |   Time (ms)   | 
|:------:|:--------:|:---------:|
|  DeepEMD-QPTH  |    -   |  >700K     |
|  DeepEMD-Sinkhorn    |    69.5 |  420     |
|  DeepEMD-IPOT     |    69.4   |  540    |
|  GEMD-M  |    72.0   |  328     |
|  GEMD-T    |   75.6 |  64     |
|  RFS     |    68.8  |  60    |
|  TSA     |    73.3  |  120    |
|  ProtoNet     |   56.1  |  45    |

**Comparisons with TSA and ProtoNet are incorporated in the modified paper (cf. Table 3e).**

Reviewer comment (Evaluation):
I notice that the proposed method performs worse than recent finetuning-based method [1,2] in SDL setting.

Author response:
Kindly note that the experimental setups of the above referenced works [1] and [2] are ***significantly different*** from ours, so  it is unfair to compare [1,2] with our methods.  

Specifically, there are several major differences in setups.

1) Backbone  is  different. The methods in [1, 2] adopt vision transformer (ViT) while we use classical convolutional networks (i.e., ResNet). 

2) Pre-training is different. [1,2] adopts self-supervised learning (SSL) based on DINO (Caro et al, ICCV 2021), while we use traditional supervised learning. 

3) Augmentations are different. [1,2] uses stronger data augmentations such as color jittering, Gaussian blur, solarization and multi-crop; in contrast, we use standard augmentations including color jittering, random horizontal clip and resized crop. 

In addition, [2] does not follow standard splits on Meta-Dataset, i.e., pretraining is performed on the whole ImageNet.  

Comparison of experimental setup

|Method|Reference \[1\]|Reference \[2\]|GEMD-T|
|:-|:-|:-|:-|
|Backbone|ViT-Tiny & ViT-Small|ViT-Small|ResNet-18|
|Pre-training|DINO-based SSL|DINO-based SSL|Supervised learning |
|Augmentation|strong|strong|standard|