Paper: BECLR: Batch Enhanced Contrastive Few-Shot Learning (ICLR-2024)

Reviewer comment (Experiments):
The evaluation should have considered the various components in separation and with respect to other methods.

Author response:
Thank you for this insightful comment, and indeed you are right, this is usually what happens in multi-component approaches. Let us first explain a few points and address your concerns. 

- First of all, please note that $\texttt{OpTA}$ **is not an existing module we adopt, but a key component** (see Key Idea II, in Section 1) and part of the novelty in the end-to-end U-FSL pipeline of $\texttt{BECLR}$. Even though in our broader impact and concluding remarks we actually propose $\texttt{OpTA}$ becoming an essential part of any FSL approach (regardless of being supervised or unsupervised), adding it on top of other prior art, cannot then be considered the same baseline anymore but **an enhanced version of it using part of our proposition**. 

- Related to this, we already have an **ablation study exactly dedicated to your point regarding component-wise analysis**. Please have a look at  Table $5$ in Section 5.2 (Ablation Studies) in the main text, where we study the robustness of $\texttt{BECLR}$ by sequentially adding essential components (including $\texttt{OpTA}$) and how this affects our end-to-end performance. This already allows us to compare against other baselines in term of performance.
    
- Next to that, we also already take the opposite route in Fig. 5 in Section 5.1 (Evaluation Results) and **demonstrate that removing $\texttt{OpTA}$ from $\texttt{BECLR}$ still gives us state-of-the-art performance** compared to prior art. In this setting, both $\texttt{BECLR}$ and competing baselines **follow identical training and testing settings**, without applying $\texttt{OpTA}$ in the inference stage, thus further ensuring fairness in our comparisons. 
    
- We will build on this direction in the following **[AQb]** to thoroughly address your interesting point.

Reviewer comment (Experiments):
It is unclear whether it is OPTA that improves performance or whether adding another baseline method might have had a similar performance; in such multi-component frameworks, this is usually an issue.

Author response:
This is a great suggestion.
- With all due respect, however, we do not think adding our innovative module on other baselines helps fairness in comparisons, but we totally agree that this substantiates the effectiveness of $\texttt{OpTA}$ and solidifies our claim on its broader impact. To address your insightful suggestion, we have put tremendous effort to conduct additional experiments, during this rebuttal period.
- We picked a suite of recent prior art in U-FSL [1-4], along with two established SSL baselines [5-6] and reproduced their results, comparing to $\texttt{BECLR}$ without $\texttt{OpTA}$, as well as per your suggestion **produced new results by plugging in $\texttt{OpTA}$** in the inference stage **for all these competing baselines**. The results (see table below) demonstrate three important points: (i) **$\texttt{BECLR w/o OpTA}$ still outperforms all those prior art**, and (ii) **$\texttt{OpTA}$ is in fact agnostic to the choice of pretraining method**, since adding it on top of prior art has considerable impact on their enhanced methodology. We argue that this is a significant point in favor of our claim that $\texttt{OpTA}$ should become an integral part of all (U-)FSL approaches. (iii) There still exists a margin between enhanced prior art and $\texttt{BECLR}$, corroborating that **it is not just $\texttt{OpTA}$ that has a meaningful effect but also $\texttt{DyCE}$ and our pretraining methodology**. Notably, in Fig.7 in the main text we already demonstrate that $\texttt{DyCE}$ and $\texttt{OpTA}$ are heavily inter-dependent and significantly affect each other's impact. To address your concern, we have included these additional empirical results and discussion in the main text in Section 5 (Evaluation Results) of the revised manuscript.

\begin{array}{l|c|cc|cc}
\hline&& \text{miniImageNet}& \text{miniImageNet} & \text{tieredImageNet}& \text{tieredImageNet}\newline \hline
\text{Method}& \text{Backbone}& \text{5 way 1 shot}& \text{5 way 5 shot}& \text{5 way 1 shot}& \text{5 way 5 shot}\newline \hline  			
\texttt{NNCLR}& \text{ResNet-18} & 63.33 ± 0.53& 80.75 ± 0.25& 65.46 ± 0.55 & 81.40 ± 0.22 \newline
\texttt{SwAV}& \text{ResNet-18} & 59.84 ± 0.52& 78.23 ± 0.26& 65.26 ± 0.53 & 81.73 ± 0.24 \newline
\texttt{CPNWCP}& \text{ResNet-18}& 53.81 ± 0.65& 69.61 ± 0.56 & 47.63 ± 0.19 & 64.18 ± 0.17\newline
\texttt{HMS}& \text{ResNet-18} 	& 58.20 ± 0.23& 75.77 ± 0.16 & 58.42 ± 0.25& 75.85 ± 0.18\newline
\texttt{PsCo}& \text{ResNet-18}& 47.24 ± 0.56& 65.48 ± 0.28 & 54.33 ± 0.54& 69.73 ± 0.49\newline
\texttt{UniSiam}& \text{ResNet-18}& 63.26 ± 0.36& 81.13 ± 0.26 & 65.18 ± 0.39 & 82.28 ± 0.29\newline
\texttt{BECLR w/o OpTA} & \text{ResNet-18} & \bf{66.14 ± 0.43}& \bf{84.32 ± 0.27} & \bf{67.86 ± 0.50} & \bf{84.16 ± 0.36}\newline \hline
\texttt{NNCLR+OpTA} & \text{ResNet-18} & 72.62 ± 0.64 & 82.19 ± 0.35& 73.16 ± 0.62& 82.82 ± 0.33 \newline
\texttt{SwAV+OpTA}& \text{ResNet-18} & 70.36 ± 0.61 & 81.68 ± 0.36 & 73.60 ± 0.63& 82.37 ± 0.34 \newline
\texttt{CPNWCP+OpTA}& \text{ResNet-18} & 60.45 ± 0.81 & 75.84 ± 0.56 & 55.05 ± 0.31& 72.91 ± 0.26 \newline
\texttt{HMS+OpTA}& \text{ResNet-18} & 69.85 ± 0.42 & 80.77 ± 0.35& 71.75 ± 0.43& 81.32 ± 0.34\newline
\texttt{PsCo+OpTA}& \text{ResNet-18} & 52.89 ± 0.71 & 67.42 ± 0.54& 57.46 ± 0.59& 70.70 ± 0.45 \newline
\texttt{UniSiam+OpTA}& \text{ResNet-18} & 72.54 ± 0.61 & 82.46 ± 0.32 & 73.37 ± 0.64& 82.64 ± 0.64 \newline
\texttt{BECLR}& \text{ResNet-18}& \bf{75.74 ± 0.62} & \bf{84.93 ± 0.33}& \bf{76.44 ± 0.66} & \bf{84.85 ± 0.37}\newline \hline
\end{array}

[1] Lu et al., Self-supervision can be a good few-shot learner, ECCV 2022 \
[2] Wang et al., Contrastive prototypical network with Wasserstein confidence penalty, ECCV 2022 \
[3] Li et al., Unsupervised few-shot image classification by learning features into clustering space,  ECCV 2022 \
[4] Jang et.al., Unsupervised Meta-learning via Few-shot Pseudo-supervised Contrastive Learning, ICLR 2023 \
[5] Dwibedi et al., With a little help from my friends: Nearest-neighbor contrastive learning of visual representations, ICCV 2021 \
[6] Caron et al., Unsupervised learning of visual features by contrasting cluster assignments, NeurIPS 2020

Reviewer comment (Experiments):
Are the results presented throughout the paper primarily based on the end-to-end process that involves all components, including OPTA?

Author response:
Yes they are, as $\texttt{BECLR}$ is an end-to-end U-FSL approach that encompasses both $\texttt{DyCE}$ and $\texttt{OpTA}$ as its key components. It should be mentioned, that this is the case for all competing baselines out there, since the U-FSL problem is solved in two sequential stages: pretraining and inference (or testing if you will), as discussed in Section 3 (Problem Statement: U-FSL). Therefore **any reported results for $\texttt{BECLR}$ and other baselines are based on the end-to-end process**. However, as we discussed earlier, to further study the robustness and versatility of $\texttt{BECLR}$, we also present its downstream performance without some of its key components, including $\texttt{OpTA}$ (see Table $5$ and Fig. $5$), also compared against the end-to-end performance of other baselines. More on this in the following.

Reviewer comment (Experiments):
To be fair, there is a need to see how other baselines work when replacing OPTA, e.g., other methods and/or linear evaluation principles for FSL. While OPTA seems to have a considerable effect, it is uncertain whether that is due to the method itself or whether other baseline methods added to this framework could have a similar effect.

Author response:
- Regarding **ablations with a different method in lieu of $\texttt{OpTA}$**, as mentioned in Sections 1 (Introduction) and 2 (Related Work) the problem of sample bias remains mostly overlooked by U-FSL baselines. This means that, to the best of our knowledge, **there is no U-FSL baseline that introduces a module, similar to $\texttt{OpTA}$, to directly target the sample bias in the inference stage of U-FSL**. This is the very reason why we do not include an ablation where we would replace our proposed $\texttt{OpTA}$ with alternative pre-existing inference modules. In contrast, in the inference stage most U-FSL baselines either directly fit a logistic classifier on top of the pretrained feature encoder e.g. [1,2] or learn a linear classification layer via fine-tuning e.g. [3,4]. That being said, the reported results for all baselines and $\texttt{BECLR}$ are based on their end-to-end performance, i.e. after both the pretraining (for learning the feature encoder) and inference stage (for predicting query images on test episodes). Finally, note that $\texttt{OpTA}$ does not introduce any additional trainable parameters nor any fine-tuning, but relies on a deterministic transportation to align the distributions of support and query sets.
    
- Indeed as you rightfully pointed out, Table 5 is an ablation study and does not concentrate on benchmarking against other baselines but delving deeper into the robustness of $\texttt{BECLR}$ and the impact of including its individual components. As such, **if we were to remove $\texttt{OpTA}$, our end-to-end performance** would be the one reported on the $5$th row of Table 5 and **would not not still go up**, since $\texttt{BECLR}$'s predictions would be prone to sample bias, as explained in Sections 1 (Introduction) and 2 (Related Work). To further clarify, rows $5$ and $6$ in Table 5 utilize the same pretrained model (which included both masking, EMA and $\texttt{DyCE}$ in the pretraining stage). Then, in the inference stage for the model without $\texttt{OpTA}$ and a given test episode, we extract query and support embeddings (using the pretrained feature encoder) and use the support set prototypes (class averages) to simply fit a logistic classifier. Instead, when $\texttt{OpTA}$ is also involved, it is utilized to transport the support set in the query set's domain, before using the modified support set prototypes for fitting the classifier. That being said, **$\texttt{BECLR}$ even without $\texttt{OpTA}$ can still be compared with and outperform the rest of U-FSL baselines** (see Fig. 5 and the Table above), since their training and evaluation settings remain identical and facilitate a fair comparison.

- Regarding the **benefit of having $\texttt{OpTA}$**, as you mentioned, we demonstrate in Table 5 how it **offers a significant performance boost for $\texttt{BECLR}$**. In fact, in the table above in **[Ab]** (and Table 4 in the revised draft), we also confirm that **$\texttt{OpTA}$ is agnostic to the choice of U-FSL pretraining method and can offer meaningful end-to-end performance gains for different U-FSL baselines**. We argue this corroborates our claim that $\texttt{OpTA}$ is a **novel add-on module** that should be an integral part (in the inference stage) of all (U-)FSL approaches, especially in the challenging lower-shot scenarios, e.g. $1$-shot. We believe that the absence of alternative (inference) modules in the U-FSL literature similar to the proposed $\texttt{OpTA}$  further highlights its impact and novelty.

Reviewer comment (Experiments):
Is there a reason why you did not consider running on the entire ImageNet?

Author response:
This is a great question! 
- To the best of our knowledge, no existing U-FSL baseline reports results on full ImageNet. FSL benchmarks define specific FSL-oriented samplers and settings which are tailored towards miniImageNet, tieredImageNet, CIFAR-FS, and FC-100 (in-domain) and miniImagenet → CDFSL and miniImagenet → CUB (cross-domain). These are **THE benchmarks adopted by $99.9$**% **of the U-FSL approaches** out there. Notably, in a separate study dedicated to SSL (and not U-FSL anymore) we are exploring the capacity of $\texttt{BECLR}$ on the entire ImageNet, but that is outside the scope of this paper. If you strongly advise, we can include some of those preliminary results in the Appendix.

Reviewer comment (Writing):
There are several typos throughout the paper — e.g., in the abstract "Critical" should be "Clinical", "downstream" should be "downstrea", and others.

Author response:
Thank you for these astute observations. Regarding the word "clinical" in the Abstract we meant that the problem of sample bias is inherent to (U-)FSL, yet we agree that critical would be a better choice of word. Regarding the typo in "downstream", this is indeed a typo that has happened on the last revision of the paper (it is not present in previous versions). Following your comment, we have carefully reviewed the revised manuscript multiple times to ensure the absence of typos.

Reviewer comment (Novelty):
The overarching idea and structure of BECLR bear a striking resemblance to PsCo, particularly when observing Figure 2. It appears that the primary distinction is the concatenation of different views of 'X' from PsCo and the addition of a dynamic clustered memory. To differentiate their work more effectively, the authors should provide a comprehensive comparison with PsCo in both the introduction and related work sections, highlighting the unique aspects of their approach.

Author response:
**[W1-a]**: Resemblance to PsCo, unique aspects of $\texttt{BECLR}$, and providing comparison with PsCo in the introduction and related work. \
**[AW1-a]**: Thank you for this helpful pointer.
- First of all, we dove deeper into PsCo and set up its pipeline for reproducing its reported results and conducting additional experiments. $\texttt{BECLR}$ and PsCo indeed share some similarities, in that both methods utilize a student-teacher momentum architecture, a memory module of past representations, some form of contrastive loss, and optimal transport (even though for different purposes). Note that none of these components are unique to neither PsCo nor $\texttt{BECLR}$ and can be found in the overall SSL literature [1-3].
- Let us now expand on their discrepancies and **unique aspects of $\texttt{BECLR}$**: (i) PsCo is based on meta learning, constructing few-shot classification (i.e., $N$-way $K$-shot) tasks, during meta-training and relying on fine-tuning for rapidly adapting to novel tasks during meta-testing. In contrast, $\texttt{BECLR}$ is a **contrastive framework based on metric/transfer learning**, which focuses on representation quality and relies on $\texttt{OpTA}$ for transferring to novel tasks (no few-shot tasks during pretraining, no fine-tuning during inference). (ii) PsCo utilizes a simple FIFO memory queue (oblivious to class-level information), while $\texttt{BECLR}$ maintains a **clustered highly-separable** (see Figure 6) **memory module** in $\texttt{DyCE}$, which (after an adaptation period) is used for sampling meaningful positives. (iii) PsCo applies optimal transport for creating pseudo-labels (directly from the unstructured memory) for $N*K$ support embeddings, in turn used as a supervisory signal to enforce consistency between support (drawn from teacher) and query (student) embeddings. In contrast, $\texttt{BECLR}$ artificially enhances the batch with additional positives and applies an instance-level contrastive loss to enforce consistency between the original and enhanced positive pairs. After each training iteration, optimal transport is applied to **update the stored clusters within $\texttt{DyCE}$ in an equipartitioned fashion** with embeddings from the current batch. (iv) Finally, $\texttt{BECLR}$ **also incorporates optimal transport (in $\texttt{OpTA}$) for aligning the distributions between support and query sets**, during inference, which shares no similarity with the pipeline of PsCo. Following your suggestion, we added further discussion on the comparison with PsCo both in Sections 2 (Related Work) and 4.1 (Unsupervised Pretraining) of the revised draft.

[1] Wang et al., Contrastive prototypical network with Wasserstein confidence penalty, ECCV 2022 \
[2] Grill et al. Bootstrap your own latent-a new approach to self-supervised learning, NeurIPS 2020 \
[3] Dwibedi et al. With a little help from my friends: Nearest-neighbor contrastive learning of visual representations, ICCV 2021 

---
**[W1-b]**: It appears that the primary distinction is the concatenation of different views of $X$ from PsCo and the addition of a dynamic clustered memory. \
**[AW1-b]**: Great remark. Let us expand further on this point:
- The reasoning behind the concatenation of different views of $X$, is to pass both views to both student/teacher networks to accommodate our symmetric contrastive loss (i.e., infoNCE loss between $Z_1^{S}$-$Z_2^{T}$ and $Z_2^{S}$-$Z_1^{T}$). The symmetricity in the loss has been shown to yield a minor performance boost [1], but does not constitute the primary distinction between PsCo and $\texttt{BECLR}$ as explained in **[AW1-a]**. The effect of adding a symmetric loss to PsCo is also discussed in greater detail in the following **[AW3]**.
- **The main novelty** and distinction lies indeed in our **dynamic clustered memory** ($\texttt{DyCE}$) and our **distribution alignment module** ($\texttt{OpTA}$), which are the two Key Ideas of our work, both contributing to the overall U-FSL performance of $\texttt{BECLR}$ (see Table 5).  

[1] Chen et. al. Exploring simple siamese representation learning, CVPR 2021

Reviewer comment (Writing):
The notation used in the algorithmic section needs elucidation. Clearly defining each symbol would make it more accessible and allow readers to follow the content with greater ease.

Author response:
**[W2]**: Elucidation of the notation in the Algorithm. \
**[AW2]**: Thank you for the constructive recommendation.
 - We did our best to define every variable in the main text and also included a step-by-step walk-through for Algorithm 2. However, we agree this could have been done better. Per your suggestion, we (i) **now include a thorough walk-through of Algorithm 1**, (ii) have made **modifications in both Algorithm 2 and its explanation** to provide more elucidation on the inner mechanics of $\texttt{DyCE}$, and (iii) made sure that **every single variable and symbol (for both algorithms) are properly defined** in the main text. These revisions are included in Section 4.1 (Unsupervised Pretraining) of the updated manuscript.

- Please also have a look at Algorithms 3 and 4, already existing in Appendix E, where we also provide PyTorch-like pseudocodes for Algorithms 1 and 2 for further elucidation.

Reviewer comment (Experiments):
The experimental section could benefit from an additional test: an evaluation of performance without merging 'X'. It would also be insightful to see results when PsCo is merged and when masking is applied, offering a more comprehensive understanding of the method's robustness and versatility.

Author response:
**[W3]**: Additional test by including/ exlcuding masking and concatenating augmented views of the input $X$. 

**[AW3]**: This is a fabulous remark and intrigued us to run new experimentation to fully address your concern here.

- Before we dive in, note that PsCo reports performance with a Conv5 backbone, even though in exploring the codebase we understood it can readily accept ResNet-18, as well. So, in the below table we present an interesting experiment. We first mark the original reported performance of PsCo with a Conv5 backbone as well as its reproduction with ResNet-18. We then modify the **loss for PsCo to be symmetric** (allowing both augmentations of $X$ to pass through both branches) similar to $\texttt{BECLR}$, (this is denoted as $\texttt{PsCo+}$). Next, we **also add patch-wise masking** to PsCo (denoted as $\texttt{PsCo++}$) and as an additional step we have **also added our novel module $\texttt{OpTA}$** on top of all these models ($\texttt{PsCo w/ OpTA}$), during inference. We observe that neither the symmetric loss nor masking are offering meaningful improvements in the performance of PsCo. On the contrary $\texttt{OpTA}$ yields a significant performance boost (up to $6.5$% in the $1$-shot setting that sample bias is most severe). This corroborates our claim that our proposed $\texttt{OpTA}$ should be considered as an add-on module to every FSL approach out there (Key Idea II). 
    
- **As an additional robustness study** we take the opposite steps for $\texttt{BECLR}$, first **removing the patch-wise masking** (denoted as $\texttt{BECLR-}$) and then **also the symmetric loss** (denoted as $\texttt{BECLR--}$), noticing that both degrade the performance slightly (see table below), which confirms our design choices to include them. Similarly, we also remove $\texttt{OpTA}$ from $\texttt{BECLR}$'s inference stage. The most important takeaway here: **the most degraded version of $\texttt{BECLR}$** ($\texttt{BECLR-- w/o OpTA}$) **still outperforms the best enhanced version of PsCo**( $\texttt{PsCo++ w/ OpTA}$), even without $\texttt{OpTA}$, which we think offers additional perspective on the advantages of adopting $\texttt{BECLR}$ over PsCo. 

- To reiterate, we believe $\texttt{BECLR}$ defines the new SoTA over ALL U-FSL baselines across all existing benchmarks dedicated to FSL (to the best of our knowledge). We have added this table and discussion in Appendix F (In-Depth Comparison with PsCo), as this study really focuses on a detailed comparison with a single baseline, but we are happy to pull this up into the main text of the revised version, in case you strongly advise to do so. We do hope this addresses your concern. 
\begin{array}{l|c|cc}
\hline
\text{Method}& \text{Backbone}& \text{5 way 1 shot} & \text{5 way 5 shot} \newline \hline
\texttt{PsCo}& \text{Conv5}& 46.70 ± 0.42& 63.26 ± 0.37 \newline
\texttt{PsCo}& \text{ResNet-18}& 47.24 ± 0.46& 65.48 ± 0.38\newline
\texttt{PsCo+}& \text{ResNet-18}& 47.86 ± 0.44& 65.95 ± 0.37\newline
\texttt{PsCo++}& \text{ResNet-18}& 47.58 ± 0.45& 65.74 ± 0.38\newline \hline
\texttt{PsCo w/ OpTA}& \text{ResNet-18} & 52.89 ± 0.61& 67.42 ± 0.51\newline
\texttt{PsCo+ w/ OpTA}& \text{ResNet-18}& 54.43 ± 0.59& 68.31 ± 0.52\newline
\texttt{PsCo++ w/ OpTA}& \text{ResNet-18}& 54.35 ± 0.60& 68.43 ± 0.52\newline \hline
\texttt{BECLR}& \text{ResNet-18}& \bf{75.74 ± 0.62} & \bf{84.93 ± 0.33}\newline
\texttt{BECLR-}& \text{ResNet-18} & 74.37 ± 0.61 & 84.19 ± 0.31\newline
\texttt{BECLR--}& \text{ResNet-18} & 73.65 ± 0.61& 83.63 ± 0.31\newline \hline
\texttt{BECLR w/o OpTA} & \text{ResNet-18}& 66.14 ± 0.43& 84.32 ± 0.27\newline
\texttt{BECLR- w/o OpTA} & \text{ResNet-18}& 65.26 ± 0.41 & 83.68 ± 0.25\newline
\texttt{BECLR-- w/o OpTA}& \text{ResNet-18} & 64.68 ± 0.44& 83.45 ± 0.26\newline \hline
\end{array}

Reviewer comment (Experiments):
How does BECLR compare to PsCo regarding the number of parameters in the model and the inference time? A direct comparison would provide clarity on the efficiency and scalability of BECLR.

Author response:
**[Q2**: Model size and inference time comparison against PsCo.  \
**[A2]**: Yet another interesting recommendation.

- Following your suggestion, during this rebuttal period **we have compared $\texttt{BECLR}$ and PsCo in terms of model size** (total number of parameters in M) **and inference times** (in sec/episode), when using the same ResNet-18 backbone architecture. The results are summarized in the table below. We notice that $\texttt{BECLR}$'s total parameter count is in fact lower than that of PsCo. Regarding inference time, $\texttt{BECLR}$ is slightly slower in comparison, yet this difference is negligible in real-time inference scenarios. This comparison has also been added in Appendix F (In-Depth Comparison with PsCo) of the revised draft, since it constitutes a comparison against a single U-FSL baseline. Please let us know if this sufficiently addresses your concern.
    
- Please also have a look at Table 13 already existing in Appendix D (Complexity Analysis), where we compare $\texttt{BECLR}$ with different SSL contrastive approaches in terms of computational complexity (model size, training time, inference time)


\begin{array}{lccc}
\hline
\text{Method} 		& \text{Backbone} 	& \text{Total Parameter Count (M)} 	& \text{Inference Time (sec/episode)} 	\newline \hline
\texttt{PsCo}            	& \text{ResNet-18}     	& 30.766                        	& 0.082                               	\newline
\texttt{BECLR}           	& \text{ResNet-18}     	& 24.199                        	& 0.216                               	\newline \hline
\end{array}

Reviewer comment (Experiments):
When pretraining with miniImagenet, which specific datasets were employed? Was the entire miniImagenet dataset utilized for this purpose?

Author response:
**[Q3]**: Data used for miniImageNet pretraining. \
**[A3]**: Thank you for this helpful comment.

- First of all, please have a look at Section B.1 (Dataset Details) already in Appendix B. Therein, we expand on the datasets that were used as part of the experimental process of $\texttt{BECLR}$, in terms of selected splits used for training, validation, and testing, number of classes and dataset size.
    
- For miniImageNet in particular, out of its $100$ classes, we select $64$, $16$, and $20$ classes for training, validation, and testing, respectively, following the **predominantly adopted settings** [1-4]. The training, validation, and testing splits and classes are mutually exclusive, as is standard practice in (U-)FSL, to test the adaptation of the model to novel tasks (unseen during training). To explicitly address your question, **when pretraining $\texttt{BECLR}$ on miniImageNet, only the training split is employed and not the entire miniImageNet dataset, nor any additional datasets.**

[1] Ravi et.al., Optimization as a model for few-shot learning, ICLR 2016 \
[2] Jang et.al., Unsupervised Meta-learning via Few-shot Pseudo-supervised Contrastive Learning, ICLR 2023 \
[3] Lu et al., Self-supervision can be a good few-shot learner, ECCV 2022 \
[4] Li et al., Unsupervised few-shot image classification by learning features into clustering space, ECCV 2022

Reviewer comment (Experiments):
SAMPTransfer (Shirekar et al 2023) is also based on membership but its performance is reported only on the miniImageNet-->CDFSL task (Table 3). It is missing from other experiments.

Author response:
**[Q1]**: SAMPTransfer performance missing. \
**[A1]**: Thanks for your insightful observation and remark. 
- First of all, please have a look at Tables $9$, $10$ and $11$, already existing in the Appendix C. **SAMPTransfer's performance on miniImageNet and tieredImageNet is reported in Table $9$ and its performance on miniImageNet → CDFSL in Table $11$** (extended versions of Tables $1$ and $3$ in the main text). 
- Regarding SAMPTransfer [1], following your suggestion, we setup the end-to-end pipeline to reproduce its results. We now know all its bells and whistles well. As we discussed already in Section 2 (Related Work to U-FSL) in the main text, SAMPTransfer's performance is hindered by its graph-based membership exploration unit and thus its performance can even degrade with larger backbones. We speculate that this is the reason why its performance is not reported on larger backbones in the original paper. To substantiate this and address your concern, we have conducted an additional experiment during this rebuttal period, comparing the performance of $\texttt{BECLR}$ and SAMPTransfer for different backbones. As seen in the table below: (i) SAMPTransfer indeed performs better with a shallower backbone (Conv4b) and (ii) $\texttt{BECLR}$ shows a superior end-to-end performance, regardless of the choice of backbone. Finally, **we have included the performance of SAMPTransfer on Table $1$ of the revised draft, following your remark**. \
Would you also advise including the below Table in the Appendix of the revised draft?

\begin{array}{lccccc}
\hline
&& \text{miniImageNet}& \text{miniImageNet}\newline \hline
\text{Method}& \text{Backbone}& \text{5 way 1 shot}& \text{5 way 5 shot}\newline \hline
\texttt{SAMPTransfer}& \text{Conv4}& 55.75 ± 0.77& 68.33 ± 0.66\newline
\texttt{SAMPTransfer}& \text{Conv4b}& 61.02 ± 1.05& 72.52 ± 0.68\newline
\texttt{BECLR}& \text{Conv4b}& \textbf{69.54 ± 0.63}& \textbf{80.61 ± 0.29}\newline \hline
\texttt{SAMPTransfer}& \text{ResNet-18}& 45.64 ± 0.73& 56.82 ± 0.65\newline
\texttt{BECLR}& \text{ResNet-18}& \textbf{75.74 ± 0.62} & \textbf{84.93 ± 0.33}\newline \hline
\end{array}


[1] Shirekar et al., Self-Attention Message Passing for Contrastive Few-Shot Learning, WACV 2023

Reviewer comment (Novelty):
Prior FSL works that are related to the distribution shift (sample bias) issue are not discussed.

Author response:
**[Q2]**: Prior FSL works that are related to the distribution shift (sample bias) are not discussed. \
**[A2]**: Great suggestion. 
- Let us first set the scene a bit better: as we articulate in the paper, the problem of sample bias (distribution shift between query and support sets) is _overlooked_ by almost every U-FSL baseline and the vast majority of the supervised FSL prior art. This is the very reason behind our Key Idea II in Section 1 (Introduction) and the proposition of the novel module called $\texttt{OpTA}$. 
- Following your suggestion, we have delved deeper in the FSL literature and found related work that attempts to alleviate the effects of sample bias. In particular, [1] proposes to enhance the support set by drawing additional base-class images via class-competitive attention maps, [2] highlights that sample bias is most severe when support embeddings are in the vicinity of the task centroid, whereas [3] utilizes a calibrated distribution for sampling more support embeddings, yet these methods explicitly rely on base-class characteristics. [4,5] follow a different perspective and tackle sample bias directly on the logistic classifier-level by using Firth bias reduction [6] techniques. These methods, however, are prone to overfitting of the classifier and pretrained encoder during fine-tuning. In contrast, our proposed $\texttt{OpTA}$ module requires no fine-tuning (and no additional trainable parameters) and does not depend on the pretraining dataset. Instead, we utilize optimal transport [7] for aligning the distributions of the _biased_ support set with that of the query set at the inference stage. **We have included this discussion in Section 2 (Related Work) of the revised version, along with further elaboration on the distribution shifts and biases that are present in the U-FSL setting.** We are happy to also include any specific references from the (U-)FSL literature you might find relevant and want us to refer to, please just advise and we will include.

[1] Chen et al., Few-Shot Learning with Part Discovery and Augmentation from Unlabeled Images, IJCAI 2021 \
[2] Xu et al., Alleviating the sample selection bias in few-shot learning by removing projection to the centroid, NeurIPS 2022 \
[3] Yang et al., Free Lunch for Few-shot Learning: Distribution Calibration, ICLR 2021 \
[4] Ghaffari et al., On the importance of firth bias reduction in few-shot classification, ICLR 2022 \
[5] Wang et al., Revisit Finetuning strategy for Few-Shot Learning to Transfer the Emdeddings, ICLR 202 \
[6] Firth, David, Bias reduction of maximum likelihood estimates. \
[7] Cuturi et al., Computational optimal transport: With applications to data science.