From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Accept: poster
metareview:_summary,_strengths_and_weaknesses: This paper addresses few-shot learning in partially-observable settings, where the goal is to meta-learn fruitful features that may be present in only some of the views. It bases its development on the product of experts to map different views to the representations with different uncertainty, capturing the contributions of different features. All of reviewers agree that the partially-observable setting seems to be novel in few-shot learning environment and the method developed here is sound. Authors did a good job in the rebuttal phase, resolving most of concerns raised by reviewers by adding baseline methods in empirical comparisons. While there is a disagreement on the issue whether the term ‘contrastive’ fits the method developed here, all reviewers agree to champion this paper.

justification_for_why_not_higher_score: Accept (poster) is the best place for this paper. 
justification_for_why_not_lower_score: Authors did a good job in responding to reviewers' concerns, leading that all reviewers suggested 'marginal accept'. In fact, this paper received the best score in my batch.
note_from_PC: if the above contains the word "oral" or "spotlight" please see: "oral" presentation means -> notable-top-5% and "spotlight" means -> notable-top-25%. As stated in our emails, we are disassociating presentation type from AC recommendations

From: ICLR.cc/2023/Conference/Paper4348/Authors
title: Author Response on Connection to Contrastive Learning
comment: Thanks for your response; we're pleased to hear that we have addressed most of your concerns.

Regarding the connection to contrastive learning, we agree that the core idea of contrastive learning is learning via comparisons, rather than the narrower scope of instance discrimination with augmentations. We highlight this distinction between the general definition of a 'view' of data (anything which partly captures the underlying information, such as a particular modality, or frame in a temporal sequence) and the specific choice of a 'view' as an augmentation in the first two sentences of our review of contrastive learning in Section 2.1. 

We believe our method is strongly related to contrastive learning not because of our use of partial views, but because of POEM's mechanism to learn how to *compare* these partial views when the features contained within these views may be distinct. For example, in our application to learning environment representations, we learn representations by positively contrasting observations of the same environment and negatively contrasting with observations of other environments. This mechanism to learn to unify these views via contrasting with views of other data items is our main contribution, so we really hope this clears up the remaining confusion.

From: ICLR.cc/2023/Conference/Paper4348/Reviewer_fe5b
title: Thanks for the response
comment: I think the responses have addressed most of my concerns, but I respectfully disagree about the authors' explanation in the connection to 'contrastive learning'. Multiple views are just one way to build positive and negative samples, other ways including co-occurrence and local-global can also serve as the way to achieve this. Building multiple views is not the core idea of contrastive learning (probably more on the energy-based model). The authors' explanation seems to narrow the scope of contrastive learning. I think the authors should find a more precise term to describe this multiple-view process.

I think my current score is still a reasonable evaluation, I will just keep it.

From: ICLR.cc/2023/Conference/Paper4348/Authors
title: Review Update Author Response
comment: Thank you for your response. We have now attached a diff version as supplementary material with changes highlighted as per your suggestion. We hope this helps with identifying the improvements that we made following the helpful feedback from the reviewers. 

We’re pleased to hear that our improvements have addressed your main concerns and thank you for increasing your recommendation. If there is any additional information we can provide to help further demonstrate the value of our work please do let us know.

From: ICLR.cc/2023/Conference/Paper4348/Reviewer_YAXs
title: Review update
comment: I believe the authors have addressed my main concerns. I have raised my score to a 6. A suggestion for the future: it would be good to mark edited portions in the manuscript by a separate text color, which would complement the OpenReview diff tool.

From: ICLR.cc/2023/Conference/Paper4348/Authors
title: Author Summary of Improvements following Reviewer Feedback
comment: We sincerely thank all of the reviewers for taking the time to review our work and for their constructive comments. In response to the feedback, we have now updated the paper to: 
1. Incorporate additional related work and references to better distinguish our contribution;
2. Include pseudocode and write our final objective more explicitly to aid reproducibility;
3. Explain our benchmarking procedure more thoroughly to highlight the strength and representativeness of our baselines;
4. Include an additional recurrent network baseline for our application to learning representations of environments;

in addition to other minor improvements to clarify the description of our contribution. We believe that our improved work provides a valuable contribution to learning with partial observations.

From: ICLR.cc/2023/Conference/Paper4348/Authors
title: Author Response to Reviewer fe5b (Part 2/2)
comment: > Although partial observation is an interesting problem, it is not commonly seen in the way PO-Meta-Dataset developed. The two experiments are more like toy problems where the feature uncertainty is manually inserted, I do not think creating the views through data augmentations used by SimCLR is the correct way to generate a realistic partial observation problem.

We fully agree that augmentations are not the ideal way to generate realistic partial observability. We were able to demonstrate that POEM provides an advantage in this partial observation setting, and hope that the method may bring even greater benefits to more complex partial observability applications in future work.

> Another way I can think of is to make use of the annotation in the CUB Bird dataset, create a support set with only some parts of a bird observed, and the query set with other parts appearing. It could be more close to the real-world scenario where objects are not always observed from the same angle, so the parts observed in the support set may be different from the parts observed in the query set for the object from the same class.

In our experimental implementation of PO-Meta-Dataset, we ensure that the support set views do not cover more than 50% in total of the area of each item from which the query view may be taken, so the task should involve some generalisation of features as you describe.

> The most common partial observation is in reinforcement learning, known as POMDP, but the authors do not explore this problem enough in this paper. 

We agree that agents learning in partially observable environments (formally POMDPs) usually take a different representation learning approach to few-shot learning as in our work, generally using a recurrent architecture to integrate observations [3] (or more simply for nearly fully-observed environments such as Atari or DeepMind Control taking a frame-stacking approach [4]). As a result, we have now added an additional reccurrent network baseline (using the hidden state of a final GRU layer as the current environment representation) into our results on the MiniGrid task in section 5.1 (and appendix A.8). We found that POEM still outperformed this recurrent approach by almost 10% in terms of observation classification accuracy, which is very promising.

> Details about the model, dataset, and environment are provided, which enhances its reproducibility, but I think the authors could give details about the whole algorithm in the appendix.

We fully agree and have now updated the theoretical formalism to more explicitly write out our final objective, and added pseudocode (both language agnostic and in PyTorch style) to the appendix to aid reproducibility.

> **Summary Of The Review**:
This paper studies an interesting problem as partial observation in few-shot learning. It derives a novel method to tackle this problem and gives an intuitive interpretation. The method is proven to outperform previous baselines on two problems. But generally, the baselines chosen in the experiment are too old and the problems designed are not realistic enough, which makes the significance of the proposed method in doubt. I think this paper is on the borderline and authors should try to add additional experiments on more realistic problems/datasets to further demonstrate the effectiveness of POEM.

Thank you again for your review. We very much hope that the additional evidence we have provided about the strength and representativeness of our baselines, alongside the additional related work comparisons and the additional recurrent network baseline we have incorporated help to assure you of the significance of our work. While we appreciate the partial observability considered in our PO-Meta-Dataset experiments is illustrative, we hope that it provides a first step towards tackling the important problem of learning from partial observations and that it inspires further work and benchmarks on this topic. We sincerely believe that our updated work provides a valuable contribution to learning with partial observations.

**References**:

[1] Baz et al., Lessons learned from the NeurIPS 2021 MetaDL challenge: Backbone fine-tuning without episodic meta-learning dominates for few-shot learning image classification, NeurIPS 2021

[2] Chen et al., MetaDelta: A Meta-Learning System for Few-shot Image Classification, AAAI 2021

[3] Hausknecht et al., Deep Recurrent Q-Learning for Partially Observable MDPs, AAAI 2015

[4] Mnih et al., Playing Atari with Deep Reinforcement Learning, NeurIPS 2013

Part 2/2

From: ICLR.cc/2023/Conference/Paper4348/Authors
title: Author Response to Reviewer fe5b (Part 1/2)
comment: Thank you for taking the time to review our work and for your helpful suggestions. In response to your comments and the comments from other reviewers, we have now updated our paper to: (1) incorporate additional related work and references to better distinguish our contribution; (2) include pseudocode and write our final objective more explicitly to aid reproducibility; (3) explain our benchmarking procedure more thoroughly to highlight the strength and representativeness of our baselines; (4) include an additional recurrent network baseline for our application to learning representations of environments; alongside other minor improvements to clarify the description our contribution.

> Strengths:
This paper identifies a novel problem that is also very important to the community, partial observation is very common in the reinforcement learning problem.
The proposed method is well-formulated and the authors also provide an intuitive interpretation of the training objective.
The experimental shows the advantage of the proposed POEM method to some extent and the ablation study further verifies the source of improvement comes from the feature uncertainty.
The newly created PO-Meta-Dataset could be useful for future studies on this problem.

We are pleased that the importance of the problem we attempt to address is appreciated, and that our experiments successfully demonstrated the source of improvement of POEM was due to feature uncertainty as intended.

> The baselines compared with POEM are all too old, the authors have mentioned many recent strong meta-learning methods in the related work part, I think the authors should also compare POEM with some of them. DeepEMD [1] is also a strong baseline.

We appreciate that the baseline algorithms we use for benchmarking are older than other more recent variants of these methods. However, we argue that these baselines, when modernised with large pre-trained backbones which are fine-tuned during meta-training (as we perform in our work), are still state-of-the-art for novel few-shot learning tasks and benchmarks. This is evidenced by, for example, the recent NeurIPS 2021 MetaDL Challenge [1] which was won by a fine-tuned Prototypical Network based approach [2]. We have now updated our implementation details to better explain our procedure, as we appreciate that our modernised training approach may not have been clear in the original version, so thank you for this feedback.

We also agree that DeepEMD is a strong few-shot learning baseline, and as a result have included an explicit comparison in our related work (section 2.4). However, as we explain in this new section, the purpose of DeepEMD is to identify informative patches of features *within* each support/query sample for fine-grained classification, whereas POEM aims to combine partial information from *across* provided support patches for comparison with a partial query. Therefore, as DeepEMD and the other methods that we mention in our related work have no explicit mechanism for handling partial observability, we believe that these additional baselines would provide no added value in analysing whether POEM performs better at few-shot learning with partial observations, which is the purpose of our experiments. 

> I do not see any point to mention contrastive learning in this paper, although there are some weak connections, like the objective in embedding-based few-shot learning and multi-views used by SimCLR, POEM is in general not strongly connected with the contrastive representation learning methods and I feel the title is kind of confusing.

We believe that there is a strong connection between our work and contrastive learning. Specifically, our work is designed to integrate multiple views of entities into a single representation by contrasting views of the same item ('positives') with views of different items ('negatives') to learn a representation of an item. We are learning how to contrastively combine representations of multiple differing views to learn a consistent representation. However, given this was not clear in our original submission, we have updated our description of the related work in contrastive learning (section 2.1) to specifically highlight this connection between our work and the relevant contrastive methods. We hope the connection with our work is now more apparent.

Part 1/2

From: ICLR.cc/2023/Conference/Paper4348/Authors
title: Author Response to Reviewer EVnw (Part 2/2)
comment: > Eq 18 and Eq 19 (from the Appendix) are introduced on page 5 without any context, I guess that’s a typo in referencing equations. 

Yes this was a typo (although it was still referencing the correct equations in the appendix) and has now been corrected.

> While the author(s) criticized prior work for using diagonal covariance (sec 2.2), base on the notations it seems that the this work also relies on diagonal covariance.

This was a mistake, so thank you for catching this! Diagonal covariance (as our work relies on) is able to capture partial observability by varying components along the diagonal. The prior work instead relied on spherical covariance, which is not able to capture partial observability since all diagonal components (uncertanties) are equal. We have corrected this in our description of the related work in Section 2.2.

> Finally, there is a lot of marginalization going on, and for real image data the probability might vary considerably in scale, causing concerns for numerical overflow issues.

In our implementation we work directly with the logarithms of the Gaussian integrals/normalisation constants and so did not have any issues with numerical overflow issues. We hope our pseudocode and updated objective demonstrate the simplicity of the implementation.

> The experiment results show solid gains over the competing baselines, but that needs to be interpreted with salt and gain: (1) these are all classical baselines and do not include more recent SOTA solutions; (2) the experiment setup differs from standard settings and it seems that no proper adjustment have been applied to the baseline solutions to accommodate the change.

We appreciate that the baseline algorithms we use for benchmarking are older than other more recent variants of these methods. However, we argue that these baselines, when modernised with large pre-trained backbones which are fine-tuned during meta-training (as we perform in our work), are still state-of-the-art for novel few-shot learning tasks and benchmarks. This is evidenced by, for example, the recent NeurIPS 2021 MetaDL Challenge [1] which was won by a fine-tuned Prototypical Network based approach [2]. We have now updated the paper to better explain our procedure, as we appreciate that our modernised training approach may not have been clear in the original version, so thank you for this feedback.

We also agree that the changed experimental setup for PO-Meta-Dataset has potential to make comparisons unfair, but we took great care to ensure our comparisons to the baseline solutions were meaningful. In particular that is why we took the time to reimplement all of our baselines with identical modernised backbone architectures, and ensured that any parameter choices were either shared where appropriate, or optimised for Meta-Dataset. Given the methods perform comparably on the standard Meta-Dataset benchmark but POEM consistently outperforms the baselines on PO-Meta-Dataset, we were able to attribute this performance improvement to POEM's ability to learn from partial observations as required by the task rather than due to tuning adjustments.

> Originality. Fair Patched-based query (partial observation) and augmented views have been explored by prior works [1-3].

Thank you for highlighting these relevant prior works, which we have now incorporated in our related work (section 2.4). As we explain in this new section, the key difference is that the purpose of these patch-based approaches is to identify informative patches of features *within* each support/query sample, whereas POEM aims to combine partial information from *across* provided support patches for comparison with a partial query.

> **Summary Of The Review**:
The current presentation lacks clarity and the claimed novelty failed to set it apart from prior arts, the experimental evidence also needs to be strengthened. Consequently, I am recommending reject at this moment. I will be happy to reconsider my evaluation if the author(s) can present strong supporting arguments or significantly improve their work during the rebuttal phase.

Thank you again for all of your many valuable suggestions to improve the quality of our paper. We very much hope that the improvements we have made to better distinguish our contribution from related work, to improve the clarity of the theoretical formalism, to demonstrate the strength and fairness of comparison with our baselines, and to improve the reproducibility of our work with clear pseudocode will allow you to reconsider your recommendation. We sincerely believe that our improved work provides a valuable contribution to learning with partial observations.

**References**:

[1] Baz et al., Lessons learned from the NeurIPS 2021 MetaDL challenge: Backbone fine-tuning without episodic meta-learning dominates for few-shot learning image classification, NeurIPS 2021

[2] Chen et al., MetaDelta: A Meta-Learning System for Few-shot Image Classification, AAAI 2021

Part 2/2

From: ICLR.cc/2023/Conference/Paper4348/Authors
title: Author Response to Reviewer EVnw (Part 1/2)
comment: Thank you for taking the time to review our work and for your very constructive suggestions. In response to your comments and the comments from other reviewers, we have now updated our paper to: (1) incorporate additional related work and references to better distinguish our contribution; (2) include pseudocode and write our final objective more explicitly to aid reproducibility; (3) explain our benchmarking procedure more thoroughly to highlight the strength and representativeness of our baselines; (4) include an additional recurrent network baseline for our application to learning representations of environments; alongside other minor improvements to clarify the description our contribution.

> The main weakness with this submission is the lack of clarity. I am unable to establish a mental model of how everything works together after a few attempts with the paper. The mathematical notations are not intuitively mapped to intuitive concepts/examples in few-shot learning. 

We are very grateful to you for highlighting the potential for misunderstanding our original submission. As a result, we have significantly revised our theoretical formalism section to address your comments below, in particular to include detailed pseudocode in the appendix (both language agnostic and PyTorch style in appendix A.3) and to write out our final objective more explicitly. We hope that the explanation of our method is now clear from the paper alone, but we also address each of your individual comments below.

> The local expert $\phi(z|x)$ is not defined when it appears in the text, and I can not find a final training objective in the main text.

We have now expanded on the definition of $\phi$ at the beginning of Section 3.1. As we now explain, $\phi$ is a conditional density: a map of a view to a distribution whose parameter values are determined by a neural network. We have also written our final objective more explicitly in the main text at the end of Section 3.1.

> In Fig 1 & 2, it seems to suggest that each view is a partial observation of the object, potentially augmented with some transformations. And it seems to suggest each sample in the support represent one particular class. So are the views for each X_m extracted from patches of all supporting images of class-m?

Yes, each view is an augmented partial observation of an object. The few-shot task in PO-Meta-Dataset is to identify which object/item a view is taken from, so each item corresponds to a class in this task. Accordingly, the views for each $X_m$ are extracted from a specific image (labelled by m), but we ensure that the support set does not cover more than 50% of the area of that image to require generalisation for comparison with the query.

>  Also Vm=dim(Xm) seems to suggest each view is one dimension of the vector X.

Here, $X_m$ was intended to refer to a mathematical set (rather than a tensor, since the data is exchangeable), so the dimension referred to the number of views contained within the set, $V_m$. However, given this notation appears to have been unclear, we have removed this notation and described this in words in the text.

> Additionally, is the query a view or a collection of views? These are all not clear from the text. 

Each query is a single view (as described in the first paragraph of section 3.1).

> If the size of support views differ for each class m, then is P(x^*|X^m) still comparable?

Yes, since these are probabilities over the same random variable ($x^*$), probability theory is clear that it doesn't matter what that probability is conditioned on (for example, consider the use of conditional entropy where we compare conditioned and unconditioned quantities). As an intuitive way to see this, since the product over support views appears in both the numerator and denominator of $P(x^*|X^m)$ for each class m (equation 4), the magnitudes of these probabilities will be comparable.

> p(z) seems to be a mixture of Gaussian on page 4, how is that tractable (are you just summing over the finite support set)? Then on page 5 the author(s) claimed that p(z) is approximated with a Gaussian, which again does not give any detail how the approximation is implemented.

$p(z)$ is indeed a mixture of Gaussians, but it cancels in the numerator and denominator, and so it only turns up in estimating the integral in the denominator (equation 6). For tractability we do approximate this with a Gaussian, but, since the prior is fairly flat in comparison to the product, experiments showed it has little effect on equation 6. Therefore it was found that it could be safely neglected (i.e. taken to the infinite variance limit) to simplify the implementation (see the footnote on page 5 and detailed description in appendix A.1). This is reflected in the pseudocode provided in appendix A.3.

Part 1/2

From: ICLR.cc/2023/Conference/Paper4348/Authors
title: Author Response to Reviewer crwa
comment: Thank you for taking the time to review our work and for your helpful suggestions. In response to your comments and the comments from other reviewers, we have now updated our paper to: (1) incorporate additional related work and references to better distinguish our contribution; (2) include pseudocode and write our final objective more explicitly to aid reproducibility; (3) explain our benchmarking procedure more thoroughly to highlight the strength and representativeness of our baselines; (4) include an additional recurrent network baseline for our application to learning representations of environments; alongside other minor improvements to clarify the description our contribution.

> Weakness Is there any bias in the creation of PO Meta-datasets. Specifically, since the other state-of-the-art models were not designed to solve the task in PO-Meta (if I understood it correctly), then are the results fair towards these models?

We agree that it is often difficult to ensure that comparisons of methods are really fair, which is why we took the time to reimplement all of our baselines with identical modernised backbone architectures, and that any parameter choices were either shared where appropriate, or optimised for Meta-Dataset. We also agree that these methods were not designed to solve the novel task in PO-Meta-Dataset, or handle partial observations of items/classes in general, but adapting these methods (Prototypical Networks most closely) to better solve this task is precisely our contribution in this work. 

Given that this concern may well be shared by other readers however, we have now incorporated additional explanation and justification for our benchmarking procedure in the section on experimental implementation details, so thank you for this feedback.

> For the agent environment, it seems like this type of representation learning is bit more forced as compared to using other techniques such as POMDPs. Is the Prototypical network the state-of-the-art for this type of learning? Is there a stronger, more realistic case for this representational learning approach in partially observable environments

We agree that agents learning in partially observable environments (formally POMDPs) usually take a different representation learning approach to few-shot learning as in our work, generally using a recurrent architecture to integrate observations [1] (or more simply for nearly fully-observed environments such as Atari or DeepMind Control taking a frame-stacking approach [2]). As a result, we have now added an additional reccurrent network baseline (using the hidden state of a final GRU layer as the current environment representation) into our results on the MiniGrid task in section 5.1 (and appendix A.8). We found that POEM still outperformed this recurrent approach by almost 10% in terms of observation classification accuracy, which is very promising.

>**Summary Of The Review**:
Overall, the paper seems to have a clear motivation backed by a nice formulation. However, since the task and the dataset is new, the significance of the comparison studies is one possible issue.

Thank you again for your review. We very much hope we have addressed your concerns about the reliability of our baseline comparisons and sincerely believe that our work provides a valuable contribution to learning with partial observations.

**References**:

[1] Hausknecht et al., Deep Recurrent Q-Learning for Partially Observable MDPs, AAAI 2015

[2] Mnih et al., Playing Atari with Deep Reinforcement Learning, NeurIPS 2013

From: ICLR.cc/2023/Conference/Paper4348/Authors
title: Author Response to Reviewer YAXs (Part 2/2)
comment: > **Reproducibility**: Code is promised upon publication. As mentioned above, it would be good to have a concise description of the objective as pseudocode to aid in long-term reproducibility.

We agree that a concise summary of the method would aid with reproducibility and clarity of the work, so thank you for this suggestion. We have now updated our paper to write out our final objective more explicitly, and to include pseudocode (both language agnostic and in PyTorch style) in appendix A.3.

> I'm also not sure that emphasizing the "contrastive" aspect of the work (e.g. in the title) is particularly important. You could say that few-shot learning has always been contrastive in the sense that different samples of the same class are "augmentations" of one another. The method doesn't seem to have particularly strong connections to standard contrastive learning objectives.

We believe that there is a strong connection between this work and contrastive learning. Specifically, this work is designed to integrate multiple views of entities into a single representation by contrasting views of the same item ('positives') with views of different items ('negatives') to learn a representation of an item. We are learning how to contrastively combine representations of multiple differing views to learn a consistent representation. However, given this was not clear in our original submission, we have updated our description of the related work in contrastive learning (section 2.1) to specifically highlight this connection between our work and the relevant contrastive methods. We hope the connection between contrastive learning and POEM is now more apparent.

> **Summary Of The Review**: Overall, the work is interesting and of good quality, but is currently mired by a few individually minor weaknesses as detailed above. As such, I currently recommend borderline rejection.

Thank you again for your clear review. We very much hope our numerous improvements to the paper alongside our comments above have addressed your concerns and that we have now better explained the value of our work for publication. We believe this work provides a valuable contribution to learning with partial observations.

**References**

[1] Baz et al., Lessons learned from the NeurIPS 2021 MetaDL challenge: Backbone fine-tuning without episodic meta-learning dominates for few-shot learning image classification, NeurIPS 2021

[2] Chen et al., MetaDelta: A Meta-Learning System for Few-shot Image Classification, AAAI 2021

Part 2/2

From: ICLR.cc/2023/Conference/Paper4348/Authors
title: Author Response to Reviewer YAXs (Part 1/2)
comment: Thank you for taking the time to review our work and for your helpful suggestions. In response to your comments and the comments from other reviewers, we have now updated our paper to: (1) incorporate additional related work and references to better distinguish our contribution; (2) include pseudocode and write our final objective more explicitly to aid reproducibility; (3) explain our benchmarking procedure more thoroughly to highlight the strength and representativeness of our baselines; (4) include an additional recurrent network baseline for our application to learning representations of environments; alongside other minor improvements to clarify the description our contribution.

> **Strengths**: The work studies partial observability, a realistic property of high-dimensional embodied observations, in the context of few-shot learning. The proposed method is reasonable and generalizes a standard method from prior work. Steps are taken to ensure fair empirical comparisons. The authors also contribute a new benchmark based on adapting an existing dataset, and experiment with an embodied toy mapping task.

We are pleased that the time we spent on careful reimplementations of our baselines to ensure fair comparisons is appreciated.

>**Weaknesses**: The baseline algorithms used in the PO-Meta-Dataset are rather weak, and it would be interesting to see to what extent more expressive models such as latent embedding optimization [A] are able to meta-learn strategies to handle the partial observability in comparison to the more explicit modeling done in POEM. Some relevant references also seem to be missing; it would also be good to discuss POEM in relation to amortized Bayesian prototype meta-learning [B], which also considers a probabilistic model of semi-parametric few-shot learning for classification. 

We appreciate that the baseline algorithms we use for benchmarking are older than other more recent variants of these methods. However, we argue that these baselines, when modernised with large pre-trained backbones which are fine-tuned during meta-training (as we perform in our work), are still state-of-the-art for novel few-shot learning tasks and benchmarks. This is evidenced by, for example, the recent NeurIPS 2021 MetaDL Challenge [1] which was won by a fine-tuned Prototypical Network based approach [2]. We have now updated our implementation details to better explain our procedure, as we appreciate that our modernised training approach may not have been completely clear in the original version, so thank you for this feedback.

We also thank you for highlighting Latent Embedding Optimisation and Amortised Bayesian Prototype Meta-Learning as related work, and agree that explicit comparison to these methods is valuable. We have therefore added discussions of both methods in our related work section. In particular, we acknowledge that LEO is an interesting unification of embedding based and optimisation based approaches and the use of a relational module to embed pairs of inputs could aid in learning features from partial observations. However, LEO, like MAML, has no mechanism to handle partially observed queries as does POEM, and crucially does not learn a unified representation of the entire support set (so could also not be used for applications such as learning environment representations). Therefore, we believe that these additional baselines (as variants of our current baselines) would provide no added value in analysing whether POEM performs better than existing methods at few-shot learning representations from partial observations, which is the purpose of our experiments.

> The authors also don't discuss whether the RoamingX datasets proposed in [C] could be suitable for evaluation, given the close relationship between embodiment and partial observability (as evidenced by the MiniGrid experiment).

Thank you for bringing our attention to Online Contextualized Few-Shot Learning, which is also an interesting application of few-shot learning that indeed brings the task closer to real world learning. The purpose of the RoamingX datasets proposed in that work is to evaluate whether context can be beneficially utilised to classify objects in an online manner. On the other hand, the purpose of our MiniGrid experiment is not to classify observations within environments, but to learn to distinguish the whole environments themselves; the aim is to demonstrate and visually evaluate an approach to unify observations of an environment at inference time into a cohesive whole.

Part 1/2

From: ICLR.cc/2023/Conference/Paper4348/Reviewer_YAXs
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposes POEM, a generalization of prototypical networks designed to handle few-shot learning when each support or query example is only a partial view/observation of the underlying concept. The derivation follows from modeling the joint distribution of support representations as the product of conditionally independent Gaussians. Prediction for a query example is done via Bayes's rule and marginalizing over the representation. For their experiments, the authors adapt a subset of Meta-Dataset so that each sample exhibits substantial partial observability. POEM is shown to improve upon baselines that do not model the partial observability. The authors also compare POEM and prototypical networks on a map reconstruction task based on the MiniGrid environment, finding that POEM enables better reconstructions faster.
strength_and_weaknesses: Strengths

The work studies partial observability, a realistic property of high-dimensional embodied observations, in the context of few-shot learning. The proposed method is reasonable and generalizes a standard method from prior work. Steps are taken to ensure fair empirical comparisons. The authors also contribute a new benchmark based on adapting an existing dataset, and experiment with an embodied toy mapping task.

Weaknesses

The baseline algorithms used in the PO-Meta-Dataset are rather weak, and it would be interesting to see to what extent more expressive models such as latent embedding optimization [A] are able to meta-learn strategies to handle the partial observability in comparison to the more explicit modeling done in POEM. Some relevant references also seem to be missing; it would also be good to discuss POEM in relation to amortized Bayesian prototype meta-learning [B], which also considers a probabilistic model of semi-parametric few-shot learning for classification. The authors also don't discuss whether the RoamingX datasets proposed in [C] could be suitable for evaluation, given the close relationship between embodiment and partial observability (as evidenced by the MiniGrid experiment).
clarity,_quality,_novelty_and_reproducibility: Clarity

It would be good to provide the concrete instantiation of POEM in algorithm box form, i.e. provide the missing link between the mathematical statement of the objective (Eqs. 8 and 9) and the reduced form afforded by simplifying assumptions, e.g. Gaussianity of each expert. 

I'm also not sure that emphasizing the "contrastive" aspect of the work (e.g. in the title) is particularly important. You could say that few-shot learning has always been contrastive in the sense that different samples of the same class are "augmentations" of one another. The method doesn't seem to have particularly strong connections to standard contrastive learning objectives.

Quality

There are several minor weaknesses as detailed above. Otherwise, the technical contribution seems correct and the empirical evaluation sound.

Novelty

There is some technical novelty in the proposed method, and some empirical novelty in the new dataset.

Reproducibility

Code is promised upon publication. As mentioned above, it would be good to have a concise description of the objective as pseudocode to aid in long-term reproducibility.
summary_of_the_review: Overall, the work is interesting and of good quality, but is currently mired by a few individually minor weaknesses as detailed above. As such, I currently recommend borderline rejection.




References

[A] Rusu et al., Meta-Learning with Latent Embedding Optimization, ICLR 2019.

[B] Sun et al., Amortized Bayesian Prototype Meta-learning: A New Probabilistic Meta-learning Approach to Few-shot Image Classification, AISTATS 2021.

[C] Ren et al., Wandering Within a World: Online Contextualized Few-Shot Learning, ICLR 2021.

correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4348/Reviewer_crwa
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper proposes a contrastive training approach for few-shot learning. The main benefit of the proposed approach is that it can perform few-shot learning in partially-observable settings.

The paper describes a formal framework for learning in partially-observable settings based on the product of experts model. Specifically, a potential function is associated with the representation learned for each view.  The factors over views are combined using a product of mixtures. To make the learning objective tractable the potential functions are assumed to be Gaussians though other types of distributions may be explored in future.

A new dataset is developed that augments an existing dataset for few-shot learning (META-DATASET). The idea is to incorporate different views with partial observability in the dataset. The proposed approach is compared on all tasks available in the META-DATASET and compared with state-of-the-art for few-shot classification. Results show that under partial observability the proposed approach outperforms existing methods while being close to state-of-the-art on datasets that o not require reasoning over partially observable views. Further, the same approach is applied to learning the representation of a grid environment based on partial observations and it is shown that the proposed approach performs well in this case.

strength_and_weaknesses: Strengths
Seems to be an intuitive idea with a nice formalism of connecting uncertainty in views to the training function
The new dataset seems to be a good contribution for others working in this area
Shows empirical performance that is quite good in different settings

Weakness
Is there any bias in the creation of PO Meta-datasets. Specifically, since the other state-of-the-art models were not designed to solve the task in PO-Meta (if I understood it correctly), then are the results fair towards these models?

For the agent environment, it seems like this type of representation learning is bit more forced as compared to using other techniques such as POMDPs. Is the Prototypical network the state-of-the-art for this type of learning? Is there a stronger, more realistic case for this representational learning approach in partially observable environments
clarity,_quality,_novelty_and_reproducibility: The writing is clear an seems well-motivated. The contributions do use existing methods (product of mixtures) but frame them in a novel manner to perform learning from multiple views. The creation of a novel dataset and a new task based on an existing dataset could also be significant in future work.
summary_of_the_review: Overall, the paper seems to have a clear motivation backed by a nice formulation. However, since the task and the dataset is new, the significance of the comparison studies is one possible issue.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4348/Reviewer_EVnw
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: +++++After Rebuttal+++++

I would like to thank the author(s) for the rebuttal and revisions made. I believe the my main concern, which is the clarity of the presentation, has been greatly improved. In response to the improvements made, I have adjusted my score. Note that I am still not fully convinced that $P(X^*|X_m)$ is comparable for different classes. 

++++++++++++++++++++

This paper develops a Bayesian formalism for few-shot learning that captures representations with different levels of uncertainty in different components. In particular, this work is motivated  by the current gap in handling views that only contains partial observations, which violates the assumption made by most existing few-shot solutions. The proposed method, Partial Observation Experts Modelling (POEM), efficiently integrates different but overlapping information from different pieces to enable better global decision making. The proposed method is extensively benchmarked on the large-scale few-shot benchmark datasets, reporting encouraging results in partial observation settings. The author also demonstrated a synthetic experiment where POEM is used to facilitate an agent exploring the environment using only partial observations. 
strength_and_weaknesses: ### Strength
* This work is very-well motivated. Given the scarcity of support examples, it is desirable to quantify the relative similarity between support and query examples with the notation of uncertainty for robust inference and better generalization. 
* In the experiment results, the proposed PEOM showed significant performance gains over the baseline solutions under the same benchmark parameters and model architectures in most cases. 

### Weakness
The main weakness with this submission is the lack of clarity. I am unable to establish a mental model of how everything works together after a few attempts with the paper. The mathematical notations are not intuitively mapped to intuitive concepts/examples in few-shot learning. The local expert $\phi(z|x)$ is not defined when it appears in the text, and I can not find a final training objective in the main text. In Fig 1 & 2, it seems to suggest that each view is a partial observation of the object, potentially augmented with some transformations. And it seems to suggest each sample in the support represent one particular class. So are the views for each X_m extracted from patches of all supporting images of class-m? Also $V^m = dim(X^m)$ seems to suggest each view is one dimension of the vector X. Additionally, is the query a view or a collection of views? These are all not clear from the text. If the size of support views differ for each class m, then is P(x^*|X^m) still comparable? p(z) seems to be a mixture of Gaussian on page 4, how is that tractable (are you just summing over the finite support set)? Then on page 5 the author(s) claimed that p(z) is approximated with a Gaussian, which again does not give any detail how the approximation is implemented. Eq 18 and Eq 19 (from the Appendix) are introduced on page 5 without any context, I guess that’s a typo in referencing equations. While the author(s) criticized prior work for using diagonal covariance (sec 2.2), base on the notations it seems that the this work also relies on diagonal covariance. Finally, there is a lot of marginalization going on, and for real image data the probability might vary considerably in scale, causing concerns for numerical overflow issues. 

clarity,_quality,_novelty_and_reproducibility: Quality. Fair
The abstract and introduction are very well-written. The author(s) clearly introduces the problem and opportunities for filling the gap in few-shot learning where the partial view and representation uncertainty are inadequately accounted for. The quality starts to degrade moving to method sections, where the problem setup is not clearly defined and the notations are confusing, there is also not enough detail on the final learning objective and how it is derived. The experiment results show solid gains over the competing baselines, but that needs to be interpreted with salt and gain: (1) these are all classical baselines and do not include more recent SOTA solutions; (2) the experiment setup differs from standard settings and it seems that no proper adjustment have been applied to the baseline solutions to accommodate the change. 

Clarity. Poor
With the current presentation, this work is not reproducible because of poor clarity. Detailed comments are given in the Weakness section above. 

Originality. Fair
Patched-based query (partial observation) and augmented views have been explored by prior works [1-3].

[1] Rethinking Generalization in Few-Shot Classification 
[2] Learning Task-aware Local Representations for Few-shot Learning
[3] Few-Shot Learning with Part Discovery and Augmentation from Unlabeled Images

summary_of_the_review: The current presentation lacks clarity and the claimed novelty failed to set it apart from prior arts, the experimental evidence also needs to be strengthened. Consequently, I am recommending reject at this moment. I will be happy to reconsider my evaluation if the author(s) can present strong supporting arguments or significantly improve their work during the rebuttal phase. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4348/Reviewer_fe5b
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper aims to tackle the partially observable few-shot learning problem where useful features are only contained in some of the views of a data instance. It develops a product of experts model to map different views to the representations with different uncertainty, which can capture the importance of different features. The proposed method is demonstrated to outperform previous few-shot learning methods on the newly created Partially Observable MetaDataset and MiniGrid environment.
strength_and_weaknesses: Strengths:
1. This paper identifies a novel problem that is also very important to the community, partial observation is very common in the reinforcement learning problem.
2. The proposed method is well-formulated and the authors also provide an intuitive interpretation of the training objective.
3. The experimental shows the advantage of the proposed POEM method to some extent and the ablation study further verifies the source of improvement comes from the feature uncertainty.
4. The newly created PO-Meta-Dataset could be useful for future studies on this problem.

Weaknesses:
1. The baselines compared with POEM are all too old, the authors have mentioned many recent strong meta-learning methods in the related work part, I think the authors should also compare POEM with some of them. DeepEMD [1] is also a strong baseline.
2. I do not see any point to mention contrastive learning in this paper, although there are some weak connections, like the objective in embedding-based few-shot learning and multi-views used by SimCLR, POEM is in general not strongly connected with the contrastive representation learning methods and I feel the title is kind of confusing.
3. Although partial observation is an interesting problem, it is not commonly seen in the way PO-Meta-Dataset developed. The two experiments are more like toy problems where the feature uncertainty is manually inserted, I do not think creating the views through data augmentations used by SimCLR is the correct way to generate a realistic partial observation problem.

   The most common partial observation is in reinforcement learning, known as POMDP, but the authors do not explore this problem enough in this paper. Another way I can think of is to make use of the annotation in the CUB Bird dataset, create a support set with only some parts of a bird observed, and the query set with other parts appearing. It could be more close to the real-world scenario where objects are not always observed from the same angle, so the parts observed in the support set may be different from the parts observed in the query set for the object from the same class.

[1] DeepEMD: Few-Shot Image Classification with Differentiable Earth Mover's Distance and Structured Classifiers. Zhang et al., CVPR 2020.

clarity,_quality,_novelty_and_reproducibility: This part is generally well-written and technically sound. But I feel the 'contrastive learning' mentioned in this paper is confusing, which seems not closely related to the method and the problem studied in this paper. The problem it studies is significant and the method is novel. Details about the model, dataset, and environment are provided, which enhances its reproducibility, but I think the authors could give details about the whole algorithm in the appendix.
summary_of_the_review: This paper studies an interesting problem as partial observation in few-shot learning. It derives a novel method to tackle this problem and gives an intuitive interpretation. The method is proven to outperform previous baselines on two problems. But generally, the baselines chosen in the experiment are too old and the problems designed are not realistic enough, which makes the significance of the proposed method in doubt. I think this paper is on the borderline and authors should try to add additional experiments on more realistic problems/datasets to further demonstrate the effectiveness of POEM. 
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold