From: ICLR.cc/2023/Conference/Paper5020/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper5020/Reviewer_spf3
title: Response to rebuttal
comment: I thank the authors for their response and appreciate their efforts to address some of my concerns. However, I am still missing some detailed comparison of total training time cost to support the claim that the method has indeed less computational cost (not the number of parameters) as is also raised by Reviewer by Sec3. More importantly, I am unfortunately still not entirely convinced about the novelty and benefits of the method over various baseline (ensemble) methods and other related work, especially after taking into account the other reviewers' comments. I, therefore, stick to my rating and encourage the authors to improve their work by including relevant comparisons of their method across all experiments (as I have raised in Q6) and with e.g. more diversity of the base model other than the random seed (as I have raised in Q5 and is also shared by reviewer N5L8) and an improved characterization of the novelty of the method.

From: ICLR.cc/2023/Conference/Paper5020/Authors
title: Response:  our goal is not to engineer representations but to investigate the limitations of the usual training process in out-of-distribution.
comment: Thanks for your response. 

In our opinion, Chowdhury et al. 2021 belong to the line of engineering "universal representation" [Li et al, 2021; Li et al, 2022; Lopes et al, 2021; Bilen & Vedaldi, 2017; Dvornik et al, 2020; wang et al, 2022] as in Q1.  Because Chowdhury et al. 2021 also use different architectures (e.g. resnet, densenet), training hyperparameters, and a different, much larger, training set (Imagenet). We should have included Chowdhury et al. 2021 into the list of references about engineering "universal representation" and we will definitely do so.

In this work, we are reluctant to engineer diversity because our goal is to show the limitations of creating representations using a single big learning episode. If we engineer diversity, we cannot talk about the properties of the learning algorithm because any improvement can be attributed to the increased diversity. Therefore the baseline that you recommend (engineer diversity) cannot serve our purpose because it prevents us from drawing our conclusion.  Note however that there is one such result in Table 3, where engineering diversity (RFC2) outperforms concatenating networks trained identically. This shows that more diversity helps, something that was already clear in the RFC paper and in all universal representation papers.  

However we also find very important to observe that we obtain substantial improvements without engineered diversity because this means that the single optimization episode approach (the dominant method) does not create the kind of "rich" representations we want.  This is a novel and important piece of knowledge was not in Chowdhury et al. 2021 because it could not have been deduced from their experiments.  We hope that you will find this bit interesting enough to revise your position.

From: ICLR.cc/2023/Conference/Paper5020/Reviewer_N5L8
title: Response to rebuttal
comment: Thank you for your answer. I have read the other reviews and your rebuttal, but I still have major concerns about this paper that have not been addressed so far. Therefore I am not changing my score.

The authors did not comment on the work of Chowdhury et al. 2021 pointed out in my first review. This work is very similar to what proposed by the authors. In particular, Chowdhury et al. 2021 show that concatenating the features of networks trained on the same dataset is very effective. What the authors have proposed is slightly different but the conclusions are very similar.

In the rebuttal the authors wrote:

*Our thesis is that in such scenarios, it is preferable to construct representations that are more "redundant" than those obtained with a single optimization episode. By "redundant", we mean representations that contain information that does not incrementally improve the training distribution performance, but can play an important role when one considers costs associated with other distributions.*

The experiments do not provide solid evidences to support this claim. A baseline that must be included is to increase the diversity of the base models (e.g. by changing the architecture, training schedule, etc) and verify if the CAT method build on top of those model is better or worst that the redundant one. Moreover, the term "redundant" here is fuzzy and needs to be grounded appropriately. Even when training with exactly the same architecture and training schedule (but different seeds) the weights of multiple models will likely cover multiple minima, therefore they are not so redundant. 

We could also think of degrees of redundancy, where the most redundant ensemble is the one built by using the same architecture, datasets, and seeds (useless as the models will be identical), and the less redundant is the ensemble built by using different architectures, trained on different datasets, with different schedules. The authors should compare ensembles along this entire spectrum to prove their point.

From: ICLR.cc/2023/Conference/Paper5020/Authors
title: New paper version 
comment: Dear Reviewers,


Thanks for your reviews! 

We want to inform you that we uploaded **a new paper version** based on your suggestions. 

We hopefully did a better job at explaining why we're doing this. 

We mark all modifications in blue. Hope it is convenient for you to read/track.

From: ICLR.cc/2023/Conference/Paper5020/Authors
title: Reference 
comment: [Chowdhury et al, 2021] Chowdhury, A., Jiang, M., Chaudhuri, S., & Jermaine, C. (2021). Few-shot image classification: Just use a library of pre-trained feature extractors and a simple classifier. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 9445-9454). 

[Li et al, 2021] Li, W. H., Liu, X., & Bilen, H. (2021). Universal representation learning from multiple domains for few-shot classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 9526-9535).  

[Li et al, 2022] Li, W. H., Liu, X., & Bilen, H. (2022). Universal Representations: A Unified Look at Multiple Task and Domain Learning. arXiv preprint arXiv:2204.02744. 

[Lopes et al, 2021] Gontijo-Lopes, R., Dauphin, Y., & Cubuk, E. D. (2021). No one representation to rule them all: Overlapping features of training methods. arXiv preprint arXiv:2110.12899. 

[Bilen & Vedaldi, 2017] Bilen, H., & Vedaldi, A. (2017). Universal representations: The missing link between faces, text, planktons, and cat breeds. arXiv preprint arXiv:1701.07275. 

[Dvornik et al, 2020] Dvornik, N., Schmid, C., & Mairal, J. (2020, August). Selecting relevant features from a multi-domain representation for few-shot classification. In European Conference on Computer Vision (pp. 769-786). Springer, Cham.  

[Kolesnikov et al, 2020] Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., & Houlsby, N. (2020, August). Big transfer (bit): General visual representation learning. In European conference on computer vision (pp. 491-507). Springer, Cham.

[Triantafillou et al, 2019] Triantafillou, E., Zhu, T., Dumoulin, V., Lamblin, P., Evci, U., Xu, K., ... & Larochelle, H. (2019). Meta-dataset: A dataset of datasets for learning to learn from few examples. arXiv preprint arXiv:1903.03096.

[wang et al, 2022] Wang, H., Frank, E., Pfahringer, B., Mayo, M., & Holmes, G. (2022). Cross-domain Few-shot Meta-learning Using Stacking. arXiv preprint arXiv:2205.05831.

From: ICLR.cc/2023/Conference/Paper5020/Authors
title: Response to reviewer N5L8
comment: We suggest the reviewer read the general response ([here](https://openreview.net/forum?id=rRgLJ8TwXe&noteId=PNfObHcS7r)) first before reading this personalized response.

---

**Q1**: The main issue of the paper is that some of the results and ideas are not new. For instance, a recent work provides results that are in line with the one presented here, meaning that concatenating the features of multiple models (pretrained on ImageNet) is very effective in cross-domain classification (Chowdhury et al. 2021). The approach itself is not new, as it has been popularized by SUR (Dvornik et al. 2020) and more recently by Wang et al. (2022). The effectiveness of distilling various models into one for cross-domain adaptation has also been empirically investigated by Li et al. (2021, 2022). More generally, the paper is aligned with the line of work on universal representations (Bilen et al, 2017). I would like to see a comment of the authors about this point, and in particular a better contextualization of their work w.r.t. the papers mentioned above.

**A**: The work on “universal representations” depends on the idea that these representations are universal because they have been obtained by training on a diversity of tasks and models hyperparameters. Here we train the same models on the same data using the same cost function, the same algorithm, and, whenever possible, the same hyper-parameters. Yet we observe a very significant effect. The existence of this effect tells us that the “universal representation” viewpoint is only a part of the story. There are other parts, such as our thesis on the (inadequate) nature of the representations constructed by a single optimization episode.
We will discuss this line of “universal representation” work [Li et al, 2021; Li et al, 2022; Lopes et al, 2021; Bilen & Vedaldi, 2017; Dvornik et al, 2020; wang et al, 2022] in the related work section. 

---

**Q2**: The deep ensemble baseline is an important one. I could not find details on how this baseline has been implemented. For instance, is the average prediction taken at the level of the logits or at the level of the softmax? 

Averaging logits, before softmax, as the deep ensemble. We also encourage the reviewer read our response to reviewer spf3’s Q7, where we explain why we should compare the performance between different representations (e.g. concatenated representations, and normally trained wide representations) instead of between different classifiers (e.g. linear classifier, ensemble classifiers, etc.)

**Q3**: In Figure 3 (top plots), the difference between the ensemble baseline (green line) and the best method (cat, blue line) seems to get smaller as the number of classes decreases. Do the authors have an explanation for this result? 

**A**: We want to answer this question in two directions: 1) the paper doesn’t focus on the comparison between ensemble and CAT, which is classifier level as the Q2 answer above. 2) Different classifiers (e.g. a large linear classifier and an ensemble n smaller classifiers) work differently depending on the number of training examples and task difficulty (e.g. number of classes). Not that the linear classifier has a larger model capacity than the ensemble classifier. The Inaturalist18 dataset contains more examples (>0.4 million) and more classes (larger task difficulty) than cifar10/100. So it is less likely to overfit and needs a larger classifier capacity. 

---

**Q4**: Experiments on the meta-learning setting (Section 7) are based on benchmarks that are becoming obsolete. The most recent papers have been using MetaDataset (Triantafillou et al. 2019) and large transfer-learning backbones as baselines (e.g. Bit, Kolesnikov et al. 2020). Testing on MetaDataset could be interesting here, as the sampler uses random-shots which is more realistic.

**A**: The miniImagenet and CUB datasets used in this work are two sub-datasets in MetaDataset.  At this point of the paper, our thesis is alredy supported quite solidly. The role of these experiments is simply to provide a validation from a slightly different angle. We believe that the results we report are sufficient for that purpose.

---

From: ICLR.cc/2023/Conference/Paper5020/Authors
title: Reference 
comment: [Li et al, 2021] Li, W. H., Liu, X., & Bilen, H. (2021). Universal representation learning from multiple domains for few-shot classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 9526-9535).  

[Li et al, 2022] Li, W. H., Liu, X., & Bilen, H. (2022). Universal Representations: A Unified Look at Multiple Task and Domain Learning. arXiv preprint arXiv:2204.02744. 

[Lopes et al, 2021] Gontijo-Lopes, R., Dauphin, Y., & Cubuk, E. D. (2021). No one representation to rule them all: Overlapping features of training methods. arXiv preprint arXiv:2110.12899. 

[Bilen & Vedaldi, 2017] Bilen, H., & Vedaldi, A. (2017). Universal representations: The missing link between faces, text, planktons, and cat breeds. arXiv preprint arXiv:1701.07275. 

[Dvornik et al, 2020] Dvornik, N., Schmid, C., & Mairal, J. (2020, August). Selecting relevant features from a multi-domain representation for few-shot classification. In European Conference on Computer Vision (pp. 769-786). Springer, Cham.  

[wang et al, 2022] Wang, H., Frank, E., Pfahringer, B., Mayo, M., & Holmes, G. (2022). Cross-domain Few-shot Meta-learning Using Stacking. arXiv preprint arXiv:2205.05831.

From: ICLR.cc/2023/Conference/Paper5020/Authors
title: Response to reviewer spf3 (2/2)
comment: **Q7**: I would have expected a more naive “ensemble” baseline where the logits of each classifier are not added but the N classifiers are combined via a naive majority vote which should be more robust. 

**A**: All these “ensemble” methods (averaging logits, averaging softmax, majority vote) and CAT method apply different classifiers on **exact same representation**, which is learned by our representation concatenation. The comparison between them only reflects the properties of classifiers, **not** the representation.

To investigate the properties of representations, we should compare the performance of classifiers **with** the concatenated representation (e.g. CAT, averaging logits, majority vote) and the performance of classifiers **without** the concatenated representation (i.e. the normally trained wide/deep networks like resnet50w4).  

---

**Q8**: The authors claim their method to be of “a far less computation burden”. To prove that I would like to see a comparison against the computation costs of training the entirety of n separately trained models and inference costs (e.g. in terms of FLOPs) over the baseline models. 

**A**: For all representations that we trained, including supervised transfer, swav self-supervised resnet50 transfer, few-shot learning, and OOD robustness, we use the same training epochs, batch size, dataset, etc. So the computation cost is proportional to the number of parameters. Considering the additional parallelization cost of training a large model on gpu clusters, the catN provides even more computation benefits than the ratio of parameter numbers.  In practice, the actual time cost also heavily depends on the implementation details. So we instead compare the number of trainable parameters.

Our additional experiments on vision transformers (ViT) show the actual inference costs are indeed proportional to the number of parameters.

From: ICLR.cc/2023/Conference/Paper5020/Authors
title: Response to reviewer spf3 (1/2)
comment: We suggest the reviewer read the general response ([here](https://openreview.net/forum?id=rRgLJ8TwXe&noteId=PNfObHcS7r)) first before reading this personalized response.

---

**Q1**: Short discussion of related work. I especially missed a more in-depth discussion about how the method relates to ensemble methods and how the method could be regarded as learning some implicit form of weighted majority voting from the combined representations using the final linear layer.

**A**: As we described in the general response, we can treat our method CAT as a “feature ensemble” rather than the normal “logits ensemble”. but, as pointed out by reviewers, the distinction can easily become fuzzy. The linear classifier trained on the concatenated representation is a superset of any weighted logit averaging (logits ensemble). It is bigger.  So it can not always be regarded as an implicit/explicit weighted logit averaging method.

---

**Q2**: Can authors give some intuition about the asymptotic behaviour of the method for concatenating N >> 1 model representations? It appears the method might become suboptimal for high N as the input dimensionality to the linear layer grows linearly which would require ever more training samples for the linear probe to avoid overfitting. 

**A**: It is true that the number of parameters grow linearly with N while the max accuracy is always less than 100%. So when N becomes very large, the cost/benefit ratio obvioulsy becomes less favorable. 

---

**Q3**: The ensemble method seems to be competitive or even superior in many experiments (the ensemble method seems to be missing for the finetuning experiments) and I would like to see more discussion on why CatN should be preferred over an ensemble method in that case (see more detailed discussion below) 

**A**: Yes. We can develop a little bit. The only difference is the construction of the last layer, either with a full linear training, or by simply averaging weights trained separately for each subnetwork. Which one works better depends on how many examples are available for the transfer task. Note that in the case of 2-stage fine tuning, we initialize the last optimization step with weights corresponding to the ensemble method. So either this overfits, or this improves the performance.

---

**Q4**: I found it a bit confusing to mostly motivate the problem of inferring (enough) features which are more relevant to transfer tasks with the optimization process. Finding these features is also very much related to the actual training data available and further assumptions on the learned models’ inductive biases (architecture, regularization, …). 

**A**: Engineering multiple source datasets, hyperparameters, and architectures is a line of work on representation learning [Li et al, 2021; Li et al, 2022; Lopes et al, 2021; Bilen & Vedaldi, 2017; Dvornik et al, 2020; wang et al, 2022]. But they need more additional prior knowledge on tasks.  Our work pursues a rich representation without these human prior knowledges, such as the choice of diverse source datasets, architectures, and hyperparameters.  

---

**Q5**: What do the authors mean by the following sentence in the intro “It is, therefore, timely to question whether the optimization of a single cost function creates and accumulates features in ways that make the most sense in this broader context”? While I very much agree with the sentence, it feels that the proposed method is actually not addressing this issue given each model backbone in the CatN method is trained using the exact same loss function and training data (merely differing in the random seed used for initialization)

**A**: Although each model backbone in the CatN method is trained using the exact same loss function, the results clearly show that these models have distinct representations (which might be redundant for the training distribution) and that combining them improves the transfer performance (meaning that they are no longer redundant for the transfer task.)

---

**Q6**: Where is the “ensemble” baseline in the finetuning experiments in Fig. 3 and Fig. 5? Why do authors take the average after softmax in section 5 but before softmax in section 6? Finally, the ensemble baselines are missing in the meta-learning and OOD robustness experiments. Why did the authors exclude the baseline there? It seems to be the most important baseline to compare. 

**A**: The “after softmax” note in our paper is a type. We always did “before softmax” ensembling. 

---

From: ICLR.cc/2023/Conference/Paper5020/Authors
title: Reference 
comment: 
[Zhai et al, 2019] Zhai, X., Puigcerver, J., Kolesnikov, A., Ruyssen, P., Riquelme, C., Lucic, M., ... & Houlsby, N. (2019). A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04867.

[Wortsman et al, 2022] Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., ... & Schmidt, L. (2022, June). Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning (pp. 23965-23998). PMLR.

[Li et al, 2021] Li, W. H., Liu, X., & Bilen, H. (2021). Universal representation learning from multiple domains for few-shot classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 9526-9535).  

[Li et al, 2022] Li, W. H., Liu, X., & Bilen, H. (2022). Universal Representations: A Unified Look at Multiple Task and Domain Learning. arXiv preprint arXiv:2204.02744. 

[Bilen & Vedaldi, 2017] Bilen, H., & Vedaldi, A. (2017). Universal representations: The missing link between faces, text, planktons, and cat breeds. arXiv preprint arXiv:1701.07275. 

[Dvornik et al, 2020] Dvornik, N., Schmid, C., & Mairal, J. (2020, August). Selecting relevant features from a multi-domain representation for few-shot classification. In European Conference on Computer Vision (pp. 769-786). Springer, Cham.  

[wang et al, 2022] Wang, H., Frank, E., Pfahringer, B., Mayo, M., & Holmes, G. (2022). Cross-domain Few-shot Meta-learning Using Stacking. arXiv preprint arXiv:2205.05831. 

[Evci et al. 2022] Evci, U., Dumoulin, V., Larochelle, H., & Mozer, M. C. (2022, June). Head2Toe: Utilizing intermediate representations for better transfer learning. In International Conference on Machine Learning (pp. 6009-6033). PMLR.

[Lopes et al, 2021] Gontijo-Lopes, R., Dauphin, Y., & Cubuk, E. D. (2021). No one representation to rule them all: Overlapping features of training methods. arXiv preprint arXiv:2110.12899. 

[doersch et al, 2020] Doersch, C., Gupta, A., & Zisserman, A. (2020). Crosstransformers: spatially-aware few-shot transfer. Advances in Neural Information Processing Systems, 33, 21981-21993.

[Roeder et al, 2021] Roeder, G., Metz, L., & Kingma, D. (2021, July). On linear identifiability of learned representations. In International Conference on Machine Learning (pp. 9030-9039). PMLR.

From: ICLR.cc/2023/Conference/Paper5020/Authors
title: Response to reviewer Bg3t (3/3)
comment: 
**Q13**: Another confusion that probably comes from the wording “construct a richer and internally redundant representation vector”. If the representation is redundant them how can it be useful? 

**A**: What is redundant for a given distribution might not be redundant when the distribution changes.  Suppose two variables x and y have high mutual information M(x,y) for a certain distribution P(x,y). If the distribution P changes, the mutual information can change in arbitrary ways.

---

**Q14**: Why are results of the baseline DISTILL5 provided in the main paper (table 3), but not the results of the proposed method?

**A**: DISTILL5, distills the representation of CAT5, is one proposed method as shown in Figure1. Furthermore, Figure 6 already shown the performance of CAT2 and CAT5.

Distill5 can be compared to single networks with a similar number of parameters (works better) or with cat5 networks that combine the same representation but have 5 times more parameters (cat5 works better).

From: ICLR.cc/2023/Conference/Paper5020/Authors
title: Response to reviewer Bg3t (2/3)
comment: **Q5**: In sec 4, why multiple training approach is deemed more reproducible than taking snapshots of the same network during training? 

**A**: As we mentioned in Figure 3 and appendix D.4, we use a large constant learning rate when taking snapshots. However, providing results that depend on an unusual learning rate schedule raises many questions about what would happen with a slightly different schedule.  This would weaken both the reproducibility and the interpretability of our results (which are not optimized for performance, but are intended to support our thesis.)


---

**Q6**: What’s the reason or evidence for saying “distillation might exclude features that could become useful when the distribution changes”? Also, why concatenating offers distribution-independent way of combining representations? 

**A**: The distillation process is distribution dependent.

---

**Q7**: In light of above question, I am curious how combining representations of different networks trained on the same data distribution can help in out-of-distribution performance. How will the models behave if the data distribution changes significantly E.g. from imagenet to satellite imagery? 

**A**: This is precisely our point. We explain that (1) despite being trained in exactly the same way, each episode provides somewhat different representations, and (2) that combining these representations benefits ood performance because, although they might be redundant with respect to the **training distribution**, they are no longer redundant when the **distribution changes**.

---

**Q8**: For transfer learning evaluation, VTAB benchmarks [5] are more widely used. I think this approach provided in this work for transfer learning should be validated on those benchmarks. 

**A**: VTAB is a limited-examples transfer learning benchmark, where each task only contains 1000 examples. But the target of this paper is **not only** the limited-example regime. So we choose transfer learning datasets in a broader regime, ranging from 1-percent cifar10 (500 examples), cifar (50000 examples), Inaturalist18 (0.4 million) to Imagenet (1.2 million). Furthermore, the transfer learning datasets used in our work are also widely used in many self-supervised transfer works, such as swav, VicReg, SimCLR, SEER, etc. So we believe that the choice of transfer learning evaluation datasets in our work is reasonable.    

---

**Q9**: When mapping from penultimate layer (d dimensional) to the last layer (k dimensional), have you accounted for increase in the number of parameters in linear layer i.e. d x k while concatenation. Concatenating n representation (i.e. n x d) would increase the parameters in linear layer by n times thus more capacity to adapt the new distribution. I would be interested to see if those added parameters confound the results. A simple test would be to concatenate representation of the same network n times and perform linear probing with that.

**A**: If we cat the same network n times, we observe no effect.  This is particularly obvious with linear probing because the optimization is convex with a well-defined optimum.

With any convex loss, an optimal solution of a linear model [w_1, ..., w_n] on the concatenation of n **identity** representation [z,...,z] is equivalent to the optimal solution of a linear model (w_1 + ,..., + w_n)/n on representation z.  So the suggestion is equivalent to learning a linear model on one learned representation, which is exactly the baseline case in our paper. 

---

**Q10**: When you take snapshots of single trained network, with relatively high step size. What is the step size and how does it compare with the step size of other models? 

**A**: Please check appendix D.4. It provides all details. 

---

**Q11**: The work only looks at convolutional networks. I think it should be be mentioned in the title of the work as the results may or may not hold for attention networks or other architectures such MLP-mixer. I would also encourage authors to validate those results on such networks. 

**A**: We mentioned the convolutional network aspect in the limitation section. We get the same conclusion on further transformer experiments too. 

---

**Q12**: Some ambiguity in the phrasing of first line of third paragraph in the intro: “out-of-distribution properties of a deep learning model can benefit from internal representations“. Should it be out-of-distribution “performance”? Or what do authors mean by out-of-distribution “properties” of a deep learning model?

**A**: Thanks. We can change that.

---

From: ICLR.cc/2023/Conference/Paper5020/Authors
title: Response to reviewer Bg3t (1/3)
comment: We suggest the reviewer read the general response ([here](https://openreview.net/forum?id=rRgLJ8TwXe&noteId=PNfObHcS7r)) first before reading this personalized response.

---

**Q1**: In the beginning, the paper argues that the optimization of a single cost function is not good enough to learn transferable representations. However, when representations of different models, that are later concatenated, are all trained with the same cost function in the proposed method.

**A**: Our general response gives a detailed explanation of this question. Here we give a short explanation. The concatenated representations, treated as a union, are **NOT** trained with a single cost function anymore. It contains multiple training episodes and each episode contains a cost function. Then the comparison between the normally trained representation (optimized by a single cost function) and the concatenated representation supports the point of this paper: “the optimization of a single cost function is not good enough to learn transferable representations” 

---

**Q2**: The related work doesn’t satisfactorily cover the previous work done in this direction. There has been some work that considers ensembling (even concatenating representations learned by) different networks [Lopes et al, 2021, doersch et al, 2020]. Kindly check [Lopes et al, 2021] for further references on the same line of work and position your work accordingly. 

**A**: There is indeed a line of work on a "universal" representation obtained by training on multiple datasets, with different hyper-parameters, and architectures [Li et al, 2021; Li et al, 2022; Lopes et al, 2021; Bilen & Vedaldi, 2017; Dvornik et al, 2020; Wang et al, 2022]. This engineered diversity is in fact a form of prior knowledge. Since our results were obtained by only changing the seeds, they show that engineering diversity is only a part of the story and that more fundamental effects are also at work. We will comnent on this line of work in the related work section.

---

**Q3**: I would encourage authors to take a look at model souping paper [Wortsman et al, 2022] as well. Although that paper looks at aggregating the weights of the networks trained with different seeds. It could be worth to make a comparison with this approach, essentially to help the community in deciding what’s the better way of utilising multiple models. 

**A**: There are two main differences between the line of weight averaging works (e.g. Model soup [Wortsman et al, 2022]) and our work that makes weight averaging fails catastrophically in our cases. 1) weight averaging is applied on multiple models that are fine-tuned from the **SAME PRETRAINED** initialization. 2) the fine-tuning hyperparameter has to be mild (e.g. a small learning rate at 1e-5 or smaller.) Both requirements are necessary for weight averaging (aka linear mode connection) because the weight averaging methods work on parameter space. This work, working on representation space, doesn’t require any of these conditions.

[Evci et al, 2022] show fine-tuning does **NOT** really construct new representations, but only “surfaces” features buried in the network. Model soup, as we described above, works exactly on the fine-tuning regime. Since we are interested in **the way representations are constructed**, we work on the pre-training phase.

---

**Q4**: It has been shown that the representations learned by networks trained with different random seeds, learn similar representations up to a linear indeterminacy [Roeder et al, 2021]. I would appreciate if the authors could shed some light on why networks should learn increasingly diverse representations while trained similarly.

**A**: We believe that our experimental results, which are easily reproduced, falsify this theory. Note that the experiments reported by [Roeder et al, 2021] do not support this Linear Identifiability hypothesis either. Both figure 3 and figure 4 (in [Roeder et al, 2021])  show the existence of substantial components of two representations that are **NOT** aligned, even after a strong SVD principal components selection. It means two representations, trained with different seeds, do **NOT** close to each other up to a linear transformation. BTW, our work is to learn useful representation for SHIFT distribution, not IID distribution. So we believe that the experiments of [Roeder et al, 2021] actually support our points.

From: ICLR.cc/2023/Conference/Paper5020/Authors
title: Reference
comment: 
[Chowdhury et al, 2021] Chowdhury, A., Jiang, M., Chaudhuri, S., & Jermaine, C. (2021). Few-shot image classification: Just use a library of pre-trained feature extractors and a simple classifier. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 9445-9454). 

[Li et al, 2021] Li, W. H., Liu, X., & Bilen, H. (2021). Universal representation learning from multiple domains for few-shot classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 9526-9535).  

[Li et al, 2022] Li, W. H., Liu, X., & Bilen, H. (2022). Universal Representations: A Unified Look at Multiple Task and Domain Learning. arXiv preprint arXiv:2204.02744. 

[Lopes et al, 2021] Gontijo-Lopes, R., Dauphin, Y., & Cubuk, E. D. (2021). No one representation to rule them all: Overlapping features of training methods. arXiv preprint arXiv:2110.12899. 

[Bilen & Vedaldi, 2017] Bilen, H., & Vedaldi, A. (2017). Universal representations: The missing link between faces, text, planktons, and cat breeds. arXiv preprint arXiv:1701.07275. 

[Dvornik et al, 2020] Dvornik, N., Schmid, C., & Mairal, J. (2020, August). Selecting relevant features from a multi-domain representation for few-shot classification. In European Conference on Computer Vision (pp. 769-786). Springer, Cham.  

[wang et al, 2022] Wang, H., Frank, E., Pfahringer, B., Mayo, M., & Holmes, G. (2022). Cross-domain Few-shot Meta-learning Using Stacking. arXiv preprint arXiv:2205.05831.

From: ICLR.cc/2023/Conference/Paper5020/Authors
title: Response to reviewer Sec3
comment: We suggest the reviewer read the general response ([here](https://openreview.net/forum?id=rRgLJ8TwXe&noteId=PNfObHcS7r)) first before reading this personalized response.

----

**Q1**: The comparison of total training time cost is not reported in the paper, although the authors claim that the proposed method has a far less computational burden. Without the time cost details, the current comparison in the paper may be unfair.

**A**: For all representations that we trained, including supervised transfer, swav self-supervised resnet50 transfer, few-shot learning, and OOD robustness, we use the same training epochs, batch size, dataset, etc. So the computation cost is proportional to the number of parameters. Considering the additional parallelization cost of training a large model on gpu clusters, the catN provides even more computation benefits than the ratio of parameter numbers.  In practice, the actual time cost also heavily depends on the implementation details. So we instead compare the number of trainable parameters.

Our additional experiments on vision transformers (ViT) show the actual inference costs are indeed proportional to the number of parameters. 

----

**Q2**: The DISTILL method seems to outperform the CAT method. I would suggest the authors provide more results of the DISTILL method to further compare the two methods: (a) Only one DISTILL model of the smallest size is reported in Section 5. Please provide more details about the DISTILL model with other sizes. (b) The DISTILL model is missing in Section 6. 

**A**: On one hand, the purpose of CAT/DISTILL/ensemble experiments in this paper is to **support our goal**, as described in the introduction. That is to determine whether the dominant approach to learn representations, namely a single optimization episode for a fixed training distribution, is still a good approach when we are dealing with multiple distributions. Our thesis is that in such scenarios, it is preferable to construct representations that are more **"redundant"** than those obtained with a single optimization episode, as described in our general response. 

CAT/ensemble concatenates the representation of multiple training episodes. DISTILL further distill the concatenated representation into a small-dimensional representation. Clearly, CAT is even simpler than DISTILL. And DISTILL is not always better than CAT as shown in Figure 3 (pink star).  Furthermore, We did not target the purely optimal performance in this work, as described in our general response. 
Thus, we provide representative DISTILL experiments in each of the supervised transfer / self-supervised transfer / few-shot / OOD domains. Note that in section 6, we actually provide the DISTILL experiments on SimSiam in Appendix Table 7.  The suggestion of applying DISTILL on other large self-supervised learning models (like RegNet256gf) might be interesting. But it is extremely expensive in practice. Even using a large cluster with 512 V100 32GB NVIDIA GPUs, distilling the large vision models in section 6 one time on Instagram1B dataset with 1 billion images will take at least several weeks. 

----


**Q3**: The comparison in Section 7 and section 8 is unfair without reporting the model sizes and training time cost.

**A**: As mentioned in Q1 and Q2 above, all experiments are used to support the paper’s goal: the importance of a rich representation and the simple ways to construct such one. On the other hand, the DISTILL models in section 7 and 8 hold exactly the same model size and dataset size as others. And thus, at least it is fair to compare DISTILL with other baseline methods. Of course, the training time of various methods (e.g. MAML, MatchingNet, ERM, CAT, DISTILL) are not aligned. Some methods, e.g. MAML, are more expensive than others. But it is not the paper’s primary goal.

----

**Q4**: Some important related work [Chowdhury et al, 2021,Li et al, 2021; Li et al, 2022; Lopes et al, 2021] are not mentioned in the paper. The authors should consider providing an in-depth introduction about the existing work.

**A**: Thanks for the suggestion. There is indeed a line of work on a "universal" representation obtained by training on multiple datasets, with different hyper-parameters, and architectures [Li et al, 2021; Li et al, 2022; Lopes et al, 2021; Bilen & Vedaldi, 2017; Dvornik et al, 2020; Wang et al, 2022]. This engineered diversity is in fact a form of prior knowledge. Since our results were obtained by only changing the seeds, they show that engineering diversity is only a part of the story and that more fundamental effects are also at work. We will comnent on this line of work in the related work section.

From: ICLR.cc/2023/Conference/Paper5020/Authors
title: General Response (2/2)
comment: The fact that our experiments give quite decent results is an accident because we merely aimed for the method that allowed us to answer our question. On the other hand, we found it important to replicate the observation on a variety of scenarios, just to make sure that our initial experiments were no fluke. We now see the **same behavior with transformers** btw.

In conclusion, because we are confident in the correctness, the novelty, and the impact of this insight, we are determined to take advantage of the reviewer's comments in order to make sure that readers get our message more clearly.

From: ICLR.cc/2023/Conference/Paper5020/Authors
title: General Response (1/2)
comment: We thank the reviewers because their comments will allow us to improve the paper. In particular, we were taken aback because all reviewers missed the point of our work. Because the reviewers are representative of the readership, we clearly must do a better job explaining that **our goal is NOT to propose yet-another-way-to-construct-ensembles**. We will certainly change our paper to make that clear and hopefully provide a new version before the next phase of the review process.

In the meantime, because of the open nature of the review process, we believe it is important to provide a more detailed clarification.
**The goal of our work**, as described in the introduction, is to determine whether the dominant approach to learning representations, namely a single optimization episode for a fixed training distribution, is still a good approach when we are dealing with multiple distributions. Our thesis is that in such scenarios, it is preferable to construct representations that are more **"redundant"** than those obtained with a single optimization episode.  By "redundant", we mean representations that contain information that does not incrementally improve the training distribution performance, but can play an important role when one considers costs associated with other distributions.

We support this theory with a collection of results obtained by simple ensembling techniques. One could say that these methods are **"feature ensembling"** techniques rather than **"output ensembling"**. but, as pointed out by reviewers, the distinction can easily become fuzzy.

Ensembling techniques are usually justified by arguing that various models have uncorrelated errors. Therefore it is customary to engineer a diversity of models. Bagging trains them with bootstrap replicas, random forests use different subsets of input variables. In fact, all of the papers listed by the reviewers (which we will of course mention in the related work section) have in common that they are ensembling models or features engineered for diversity (**different** training sets, different architectures, different training protocols, etc.).

Note that engineering this diversity can be construed as introducing a form of **prior information about the task(s)**.  Note also that because of this engineered diversity, very little can be said about the properties of the algorithm that learns features. Instead, it is assumed that the algorithm returns diverse features because it is run in diverse conditions.

This is not the case in our experiments. All training episodes target the **same** distribution with the same architecture, the same cost function and, at least in the supervised transfer / meta-learning / OOD / parts of self-supervised transfer experiments, exactly the same hyperparameters.

This is certainly not the best way to construct ensembles, but this is a way that allows us to support our thesis.

Consider first the left plot (tall plot) in figure 3 which shows the in-distribution performance. Observe how the catN curve asymptotes below the performance of the widest single network. Clearly our ensembling technique is substandard for iid performance. Meanwhile, we see a very different story in transfer learning performance (small plots) where the catN models clearly outperform the wide models. How to explain this?

Our explanation, also supported by the preliminary experiments of section 2, relies on the fact that the only diversity in our ensemble comes from the dynamics of the training process: *"Although the optimization process has no reason to accumulate features that no longer help the training objective, these same features might have been found useful when the network representation was still wanting. Therefore we posit that we can obtain substantially different representations by simply performing multiple training episodes with different seeds, that is, different initial weights and different data orderings"* -- and we also observe that having all of them at once substantially benefits the transfer learning performance (certainly more than it benefits the iid performance.)

Our conclusion is that, in a world of changing distributions, learning representations with a single optimization episode is not a good idea, because better representations can be constructed with substantially less computational effort. Note that the method we used is certainly not the most efficient one. We chose this one because it made the result as clear as possible. We only briefly mention the "snapshot" method which is potentially a lot less costly. 

We believe that this conclusion is a piece of knowledge far more consequential than yet-another-way-to-construct-not-so-good-ensembles because it can lead to **more interesting representation learning algorithms** which may or may not be ensembles.

From: ICLR.cc/2023/Conference/Paper5020/Reviewer_Sec3
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper aims at learning representations which can perform well when transferred to a different task. It presents a simple method called CAT that combines feature extractors learned during multiple training episodes. Extensive experiment results show that CAT can match or exceed the performance of baseline methods.
strength_and_weaknesses: **Strength**
1. The target problem is important and the presented method is simple to implement. 
2. Extensive experiments are conducted to compare the method with baselines.

**Weaknesses**
1. The comparison of total training time cost is not reported in the paper, although the authors claim that the proposed method has a far less computational burden. Without the time cost details, the current comparison in the paper may be unfair.

2. The DISTILL method seems to outperform the CAT method. I would suggest the authors provide more results of the DISTILL method to further compare the two methods:
(a) Only one DISTILL model of the smallest size is reported in Section 5. Please provide more details about the DISTILL model with other sizes.
(b) The DISTILL model is missing in Section 6.

3. The comparison in Section 7 and section 8 is unfair without reporting the model sizes and training time cost.

4. Some important related work (as listed below) are not mentioned in the paper. The authors should consider providing an in-depth introduction about the existing work.

[1] Chowdhury, et al. Few-shot image classification: Just use a library of pre-trained feature extractors and a simple classifier. ICCV 2021.

[2] Li et al. Universal representation learning from multiple domains for few-shot classification. ICCV 2021.

[3] Li et al. Universal Representations: A Unified Look at Multiple Task and Domain Learning. 2022.

[4] Lopes et al. No One Representation to Rule Them All: Overlapping Features of Training Methods, ICLR 2022.

**Minor problem**

The term, ERM, is used in the paper without any definition.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to implement. Given the current version, the contribution is relatively incremental.
summary_of_the_review: The target problem of the paper is interesting. My major concern lies at the empirical comparisons and the related work part.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5020/Reviewer_Bg3t
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper presents and evaluates a simple representation learning method, where embeddings from multiple networks, trained separately with different random seeds, are concatenated to get better transfer learning performance. The authors provide analysis on supervised, self-supervised learning and few-shot transfer learning scenarios.
strength_and_weaknesses: ### Strengths
- The paper argues for a simple approach for utilizing representations of learned networks i.e. concatenating embeddings of the penultimate layer.

### Weaknesses and Questions
- My main concern is limited novelty of the work. Even though it's a simple proposition, the analysis (either theoretical or empirical) require more rigour. I will expand on this below.
- In the beginning, the paper argues that the optimisation of a single cost function is not good enough to learn transferable representations. However, when representations of different models, that are later concatenated, are all trained with the same cost function in the proposed method.
- The related work doesn’t satisfactorily cover the previous work done in this direction. There has been some work that considers ensembling (even concatenating representations learned by) different networks [2, 4]. Kindly check [2] for further references on the same line of work and position your work accordingly.
- I would encourage authors to take a look at model souping paper [3] as well. Although that paper looks at aggregating the weights of the networks trained with different seeds. It could be worth to make a comparison with this approach, essentially to help the community in deciding what’s the better way of utilising multiple models.
- It has been shown that the representations learned by networks trained with different random seeds, learn similar representations up to a linear indeterminacy [1]. I would appreciate if the authors could shed some light on why networks should learn increasingly diverse representations while trained similarly. 
- In sec 4, why is multiple training approach is deemed more reproducible than taking snapshots of the same network during training?
- What’s the reason or evidence for saying “distillation might exclude features that could become useful when the distribution changes”? Also, why concatenating offers distribution-independent way of combining representations?
- In light of above question, I am curious how combining representations of different networks trained on the same data distribution can help in out-of-distribution performance. How will the models behave if the data distribution changes significantly E.g. from imagenet to satellite imagery?
- For transfer learning evaluation, VTAB benchmarks [5] are more widely used. I think this approach provided in this work for transfer learning should be validated on those benchmarks.
- When mapping from penultimate layer (d dimensional) to the last layer (k dimensional), have you accounted for increase in the number of parameters in linear layer i.e. d x k while concatenation. Concatenating n representation (i.e. n x d) would increase the parameters in linear layer by n times thus more capacity to adapt the new distribution. I would be interested to see if those added parameters confound the results. A simple test would be to concatenate representation of the same network n times and perform linear probing with that.
- When you take snapshots of single trained network, with relatively high step size. What is the step size and how does it compare with the step size of other models?
- The work only looks at convolutional networks. I think it should be be mentioned in the title of the work as the results may or may not hold for attention networks or other architectures such MLP-mixer. I would also encourage authors to validate those results on such networks.


[1] https://proceedings.mlr.press/v139/roeder21a.html
[2] https://openreview.net/forum?id=BK-4qbGgIE3
[3] https://arxiv.org/abs/2203.05482
[4] https://arxiv.org/abs/2007.11498
[5] https://ai.googleblog.com/2019/11/the-visual-task-adaptation-benchmark.html
clarity,_quality,_novelty_and_reproducibility: Clarity of the work is not up to the mark. There are some phrases which don’t convey the message clearly. I have put some of the examples below and  I would encourage authors to revise the manuscript.

- Some ambiguity in the phrasing of first line of third paragraph in the intro: “out-of-distribution properties of a deep learning model can benefit from internal representations“. Should it be out-of-distribution “performance”? Or what do authors mean by out-of-distribution “properties” of a deep learning model?
-  Another confusion that probably comes from the wording “construct a richer and internally redundant representation vector”. If the representation is redundant them how can it be useful?
-  Why are results of the baseline DISTILL5 provided in the main paper (table 3), but not the results of the proposed method?
summary_of_the_review: Objectively, the paper lacks rigour. There are no theoretical justifications of the proposed schema and the empirical justifications are limited. I would encourage authors to try out their method on more rigorous benchmarks for transfer learning such VTAB datasets.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper5020/Reviewer_spf3
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper proposes a method to improve the transfer of pretrained models to novel tasks or data distributions by using a method called CatN. Specifically, the method constructs a bigger representation by combining the representations of the last layer of N separately trained models that only differ in their initialization. The authors evaluate and validate their method against a variety of baseline methods on a comprehensive set of experiments covering supervised and self-supervised transfer learning as well as few-shot learning and OoD generalization tasks.

strength_and_weaknesses: Strengths
- The proposed method is simple, broadly applicable and straightforward to implement.
- The paper is clearly written and easy to follow.
- The authors conduct very extensive experiments.
- Explicit discussion of the remaining limitations.
- The method is applicable across different task settings.

Weaknesses
- Short discussion of related work. I especially missed a more in-depth discussion about how the method relates to ensemble methods and how the method could be regarded as learning some implicit form of weighted majority voting from the combined representations using the final linear layer.
- Can authors give some intuition about the asymptotic behaviour of the method for concatenating N >> 1 model representations? It appears the method might become suboptimal for high N as the input dimensionality to the linear layer grows linearly which would require ever more training samples for the linear probe to avoid overfitting.
- The ensemble method seems to be competitive or even superior in many experiments (the ensemble method seems to be missing for the finetuning experiments) and I would like to see more discussion on why CatN should be preferred over an ensemble method in that case (see more detailed discussion below)
- I found it a bit confusing to mostly motivate the problem of inferring (enough) features which are more relevant to transfer tasks with the optimization process. Finding these features is also very much related to the actual training data available and further assumptions on the learned models’ inductive biases (architecture, regularization, …). 
- What do the authors mean by the following sentence in the intro “It is, therefore, timely to question whether the optimization of a single cost function creates and accumulates features in ways that make the most sense in this broader context”?  While I very much agree with the sentence, it feels that the proposed method is actually not addressing this issue given each model backbone in the CatN method is trained using the exact same loss function and training data (merely differing in the random seed used for initialization)

For the experiments, I would like to see the following addressed:

- Where is the “ensemble” baseline in the finetuning experiments in Fig. 3 and Fig. 5? Why do authors take the average *after* softmax in section 5 but *before* softmax in section 6? Finally, the ensemble baselines are missing in the meta-learning and OOD robustness experiments. Why did the authors exclude the baseline there? It seems to be the most important baseline to compare.
- I would have expected a more naive “ensemble” baseline where the logits of each classifier are not added but the N classifiers are combined via a naive majority vote which should be more robust.
- The authors claim their method to be of “a far less computation burden”. To prove that I would like to see a comparison against the computation costs of training the entirety of n separately trained models and inference costs (e.g. in terms of FLOPs) over the baseline models.

clarity,_quality,_novelty_and_reproducibility: The paper is mostly well-written and easy to follow and sufficient information about the experimental details and setup is provided. The presentation of the work could be a bit improved by better highlighting the best-performing models and grouping results according to similar-size models to aid comparison.

summary_of_the_review: Overall, I found the paper to be well-written, introducing a simple, yet broadly applicable method across different transfer scenarios. Based on several weaknesses and concerns regarding the method's effectiveness compared to especially the ensemble baseline I lean towards rejecting the paper.

correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5020/Reviewer_N5L8
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper investigates representation learning for shifting tasks and distributions. In particular it introduces two methods to combine the features learned during different training episodes, showing that this combined representation is richer than the one obtained by training on a single episode. One of the main points of the paper is that assembling the representations obtained by training smaller networks outperforms training an equivalent network from scratch. Claims are supported by empirical results on self-supervised learning, meta-learning, and out-of-distribution generalization.
strength_and_weaknesses: 
Strength
--------

- Representation learning for cross-domain settings is an important and relevant topic.

- There is a large set of experiments, the authors investigated the problem from different angles.

Weaknesses
----------

1) The main issue of the paper is that some of the results and ideas are not new. For instance, a recent work provides results that are in line with the one presented here, meaning that concatenating the features of multiple models (pretrained on ImageNet) is very effective in cross-domain classification (Chowdhury et al. 2021). The approach itself is not new, as it has been popularized by SUR (Dvornik et al. 2020) and more recently by Wang et al. (2022). The effectiveness of distilling various models into one for cross-domain adaptation has also been empirically investigated by Li et al. (2021, 2022). More generally, the paper is aligned with the line of work on universal representations (Bilen et al, 2017). I would like to see a comment of the authors about this point, and in particular a better contextualization of their work w.r.t. the papers mentioned above.

2) The deep ensemble baseline is an important one. I could not find details on how this baseline has been implemented. For instance, is the average prediction taken at the level of the logits or at the level of the softmax?

3) In Figure 3 (top plots), the difference between the ensemble baseline (green line) and the best method (cat, blue line) seems to get smaller as the number of classes decreases. Do the authors have an explanation for this result?

4) Experiments on the meta-learning setting (Section 7) are based on benchmarks that are becoming obsolete. The most recent papers have been using MetaDataset (Triantafillou et al. 2019) and large transfer-learning backbones as baselines (e.g. Bit, Kolesnikov et al. 2020). Testing on MetaDataset could be interesting here, as the sampler uses random-shots which is more realistic.


References
----------

Bilen, H., & Vedaldi, A. (2017). Universal representations: The missing link between faces, text, planktons, and cat breeds. arXiv preprint arXiv:1701.07275.

Chowdhury, A., Jiang, M., Chaudhuri, S., & Jermaine, C. (2021). Few-shot image classification: Just use a library of pre-trained feature extractors and a simple classifier. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 9445-9454).

Dvornik, N., Schmid, C., & Mairal, J. (2020, August). Selecting relevant features from a multi-domain representation for few-shot classification. In European Conference on Computer Vision (pp. 769-786). Springer, Cham.

Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., & Houlsby, N. (2020, August). Big transfer (bit): General visual representation learning. In European conference on computer vision (pp. 491-507). Springer, Cham.

Li, W. H., Liu, X., & Bilen, H. (2021). Universal representation learning from multiple domains for few-shot classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 9526-9535).

Li, W. H., Liu, X., & Bilen, H. (2022). Universal Representations: A Unified Look at Multiple Task and Domain Learning. arXiv preprint arXiv:2204.02744.

Triantafillou, E., Zhu, T., Dumoulin, V., Lamblin, P., Evci, U., Xu, K., ... & Larochelle, H. (2019). Meta-dataset: A dataset of datasets for learning to learn from few examples. arXiv preprint arXiv:1903.03096.

Wang, H., Frank, E., Pfahringer, B., Mayo, M., & Holmes, G. (2022). Cross-domain Few-shot Meta-learning Using Stacking. arXiv preprint arXiv:2205.05831.
clarity,_quality,_novelty_and_reproducibility: The paper can be improved from the point of view of clarity. A better discussion of related work is needed. It would be helpful if the main contributions are explicitly stated upfront in the introduction. The quality of the paper is sufficient. Details on how to reproduce the experiments seem sufficient, code will be released.
summary_of_the_review: The paper provides a wide set of experiments to support its claims. However, some of the results and the ideas are not new, this significantly reduces the novelty of the paper.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
details_of_ethics_concerns: None.
recommendation: 3: reject, not good enough